Background: There is an urgent and unmet need for accessible and credible health information within the transgender and gender-diverse (TGD) community. Currently, TGD individuals often seek and must find relevant resources by vetting social media posts. A resource that provides accessible and credible health-related resources and content via a mobile phone app may have a positive impact on and support the TGD population.
Objective: COVID-19 stay-at-home orders forced a shift in the methods used in participatory design. In this paper, we aimed to describe the web-based participatory methods used to develop the Transgender Health Information Resource. We also described and characterized the web-based engagement that occurred during a single session of the overall design process.
Methods: We planned and conducted web-based design sessions to replace the proposed in-person sessions. We used web-based collaborative tools, including Zoom (Zoom Video Communications), Mural (Mural), REDCap (Research Electronic Data Capture; Vanderbilt University), and Justinmind (Justinmind), to engage the participants in the design process. Zoom was used as an integrated platform for design activities. Mural was used to perform exercises, such as free listing, brainstorming, and grouping. REDCap allowed us to collect survey responses. Justinmind was used to create prototypes that were shared and discussed via Zoom. Recruitment was led by one of our community partners, One Colorado, who used private Facebook groups in which web-based flyers were dispersed. The design process took place in several workshops over a period of 10 months. We described and characterized engagement during a single design session by tracking the number of influential interactions among participants. We defined an influential interaction as communication, either verbal or web-based content manipulation, that advanced the design process.
Results: We presented data from a single design session that lasted 1 hour and 48 minutes and included 4 participants. During the session, there were 301 influential interactions, consisting of 79 verbal comments and 222 web-based content manipulations.
Conclusions: Web-based participatory design can elicit input and decisions from participants to develop a health information resource, such as a mobile app user interface. Overall, participants were highly engaged. This approach maintained the benefits and fidelity of traditional in-person design sessions, mitigated deficits, and exploited the previously unconsidered benefits of web-based methods, such as enhancing the ability to participate for those who live far from academic institutions. The web-based approach to participatory design was an efficient and feasible methodological design approach.
Transgender and Gender-Diverse Health and Medical Information Needs
Transgender and gender-diverse (TGD) individuals (defined as people whose current gender is different from that assigned at birth, including, but not limited to, nonbinary, queer, and gender nonconforming people, hereafter shortened to TGD) face health disparities including high degrees of stigma and discrimination from providers and health care systems [- ]. Three-fourths of TGD individuals report negative experiences with the health care system [ ]. TGD individuals report difficulties finding and accessing TGD-competent health care professionals [ - ], securing insurance coverage for their health care needs [ - ], and finding health care professionals who are sensitive to the needs of the TGD population [ ]. Transgender individuals experience stigma and discrimination across the social determinants of health, including bullying in schools, lack of stable income, and quality housing [ - ]. Moreover, TGD individuals often have to manage chronic stress owing to traumatic experiences over their life course [ , ].
Owing to stigma and discrimination when seeking health care, TGD community members often turn to health and medical information on the web [, ]. Documented examples of insensitive health care include gender insensitivity in which individuals were misgendered (using “he” when a “they” pronoun was requested) or forced care where some patients felt they were forced to do unnecessary examinations or dismissed as “psych cases” [ ]. A growing body of literature focuses on the TGD community and their health information–seeking behavior on the web [ , , ]. A study found that gender transition mental health message boards are popular, especially on the Tumblr platform [ ]. In another study, younger TGD individuals used various web-based platforms to explore transgender, nonbinary, and gender-diverse identities and to find support networks [ ]. There is evidence that social media is a key resource relied upon by the TGD community to obtain health and medical information [ ]. The affordances of social media provide a network for peer-to-peer, emotional, appraisal, and informational support [ ].
Considering the known difficulties in seeking and determining the credibility of web-based transgender health information and the current lack of transgender-specific materials on the web [, ], we aimed to create a health information resource to support the TGD population. For this project, credible information was defined as information created or disseminated by clinicians and organizations, such as the Trevor Project [ ], with expert knowledge about care specific to the needs of transgender individuals, and reputable sources of health information such as MEDLINEPlus [ ]. Published literature highlights the lack of credible web-based information resources dedicated to the needs of TGD individuals [ , , ]. Digital tools might be important for TGD health self-management, but they are currently underutilized [ ].
To address this need, we developed the Transgender Health Information Resource (TGHIR). We chose a mobile app (Android and iOS mobile operating system) user interface as the optimal method of delivery because of mobile phone ownership—85% of Americans own a smartphone —and mobile phones support anywhere and anytime access to information.
Participatory Design and Web-Based Participatory Design
Participatory design has proven to be successful in designing mobile health resources. The benefits of this approach include engaging co-designers (henceforth, participants) selected to be representative of the community of intended end users to thoroughly explore and prioritize target audience needs [- ]. Researchers and participants can collaborate and design [ ] interfaces that make information accessible. Representation in the design process is helpful because researchers often do not understand how others are affected by technology performance [ ].
Leveraging the tacit knowledge and lived experiences [, ] of individuals from a community helps to understand how everyday tasks are conceptualized, approached, and completed. This process is typically made easier through in-person design sessions. The benefits of in-person collaboration include rapport building [ ], a sense of ownership from participants who co-design the system [ ], shared values on which design facilitators can build energy within the participatory design methodology [ ], as well as perceiving subtle cues of interpersonal communications such as facial expressions and body language [ ]. During in-person design sessions, brainstorming and drawing activities are conducted using tangible tools, including butcher paper (a type of heavy paper hung on walls to collect and record ideas) and sticky notes for rearranging the linkages, groupings, and prioritization of specific ideas, allowing for rapid iteration. These activities, and their necessary physical tools, support collaboration and inspire meaningful dialogue between participants and researchers [ ].
Approaches to evaluating participatory design have focused on the processes deployed, effects on designers, and outcomes such as satisfaction and empowerment [, ]. Evaluation of the design process and decision-making have included the collection of qualitative data from end users and system developers to determine the effectiveness of the decision-making [ ]. When the focus was on the effect of the design process on participants, evaluators concentrated on participant experiences through interviews [ ]. Outcomes, such as participant gains, can also be evaluated, by assessing participant experiences through interviews or surveys to measure how the design product benefited them and if that benefit lasted [ ]. The methodological discipline of participatory design is based on meaningful communication between participants and researchers and allows decision-making by participants, such as how health information is accessed and displayed.
The TGHIR design process coincided with the early months of the COVID-19 pandemic, which forced a transition from in-person to web-based design sessions. Although remote collaboration for design-based user experience interviews was becoming more common [, ], the use of a suite of web-based tools to support web-based design collaboration has been less documented, especially in the TGD community [ ]. Early participatory design studies [ - ] focused on how communication technologies such as email and websites could organize the web-based design process. The effective use of these communication tools, in combination with a shared web-based space hosted on the internet where design could happen synchronously or asynchronously, led to the advancement of how participatory design could be implemented. The addition of shared web-based creative spaces supported participation among remote team members while requiring fewer resources.
Web-based participatory design to support community-driven development of products on an asynchronous discussion forum has proven to be successful. Research conducted by Hess and Pipek  indicated that engaging web-based communities to support community-driven development of consumer software is possible, especially if the work is intrinsically fun. The authors found that participants on the web could contribute to the design process of a software system. However, the project began to feel unpaid by some members. A power balance between participants on the web and professional designers was observed and influenced the decisions made for the system. The findings of this study suggested that the responsibilities of participants on the web should be limited to distinct use cases so the development process is not dominated by the most engaged volunteers who might have affordances, such as levels of experience or more time to participate, which may allow greater influence on design decisions. Our proposed web-based participatory design addressed some of these challenges.
Participatory design has been used by other research teams to support the design of health information resources [- ]. This project was potentially the first to engage TGD participants exclusively in a web-based participatory design process. The web-based method of engagement might safeguard privacy and safety and allow greater involvement of TGD individuals in research. The overall objective of this paper was to describe our web-based participatory methodology and engagement evaluation for design session 1.
The setting was the University of Colorado Anschutz Medical Campus, an urban academic medical center, in collaboration with the University of Colorado Integrated Transgender Clinic, and One Colorado, the state’s leading advocacy group for LGBTQ (lesbian, gay, bisexual, transgender, and queer) persons. The design and development of the TGHIR were guided by a participatory process to ensure that the final TGHIR design would serve the health needs and goals of the TGD community. The process involved deploying a series of iterative methods to explore the use context and needs of end users. We described and characterized web-based methods and engagement, including qualitative insights from participants on the design of the TGHIR and the number of influential verbal and web-based participant interactions.
There were 3 groups involved in the design of the TGHIR: researchers, advisers, and participants. Researchers were responsible for facilitating the design sessions and implementing the decisions made in partnership with the advisers and participants as well as the agile  development of the resource. Advisers were partners [ , ] in the design process and provided feedback on participant engagement strategies, insights from design sessions, and the development process. Participants were responsible for generating ideas and making decisions on how the TGHIR should be designed, features to include, and wireframing.
Participants had to be aged 18 years and meet one of the following two inclusion criteria: (1) self-identified as TGD at any point in their journey or (2) be parents or guardians of a TGD youth.
Recruitment took place during the COVID-19 stay-at-home orders from the State of Colorado. Strategies for recruitment included posting web-based flyers in TGD Facebook groups and delivery through listservs. The outreach was led by a local TGD community partner, One Colorado . We conducted a thorough eligibility call over the Zoom videoconferencing platform (Zoom Video Communications) with each participant to assess their interest in transgender health and how they would like to contribute, which allowed the researcher to confirm the intent of their interest and that the participants could connect to Zoom [ ].
Community Advisory Board members, known as advisers, were nominated, after funding for the project was obtained, by partners and allies in the transgender community, One Colorado, and the Integrated Clinic at the CU Anschutz Medical Campus. The process of adviser selection included a nomination phase and an interview phase to assess whether the nominee’s goals and motivation fit with the objectives of the project. All nominated individuals were onboarded. Community Advisory Board membership included 8 TGD individuals, 3 parents of TGD adolescents, 1 advocate from One Colorado, 4 health care providers who served the TGD population, 1 library scientist, and 4 research staff, totaling 18 members (3 advisers were in 2 of the categories described above). Advisers attended a 4-hour in-person kickoff meeting and eight 1-hour web-based meetings. The topics of the 8 web-based meetings included purpose-to-practice exercise, identifying credible health information, asynchronous work group planning, a discussion on the Black Lives Matter movement in the summer of 2020, a review of health care provider resources, focus group and design session data discussion, TGHIR app demo feedback, usability testing data discussion, and a celebration to acknowledge what we achieved together. Each adviser was compensated US $50 per hour for attending the meetings.
Participatory Design Sessions
A web-based design approach required adapting in-person participatory design approaches to understand the targeted end users, the tasks end users were attempting to complete, and the environment in which participants completed the tasks . The three stages of the TGHIR participatory design approach were adapted from Spinuzzi [ ] and included (1) initial exploration of end-user needs, (2) discovery processes of prioritization and ideation on potential outcomes, and (3) prototyping. As a research team, we integrated these stages into a larger design process to access the opinions and experiences of participants on behalf of the targeted end users. A total of 4 distinct design sessions were conducted as part of the TGHIR design process ( ). We described the methodology implemented in these sessions, including how the methods were adapted to web-based interactions and the degree of participation in the “Ladder of Participation” column [ , ], a theoretical construct that described the range of co-designer participation from low to high.
Several web-based tools were used for web-based participatory design session implementation, including Zoom , Mural (Mural) [ ], REDCap (Research Electronic Data Capture; Vanderbilt University) [ ], and Justinmind (Justinmind) [ ]. Zoom was used primarily to allow all participants to meet, with audio and video capabilities using a computer or tablet and computing functionality in a common web-based space. Mural was used as the collaborative workspace for the design sessions; participants were asked to create free accounts and were then sent a link to collaborate in a Mural workspace. REDCap was used for web-based surveys and Justinmind software was used for wireframing and prototyping.
|Design session and goals (research participants)||Participatory design stage adapted from Spinuzzi ||Ladder of participation||Web-based tools||Planned in-person tools|
|Session 1: Exploration of potential features (n=4)||Stage 1: initial exploration of end-user needs||Consultants: participant feedback on features||Mural and Zoom||Butcher paper, sticky notes, markers, and bullseye visualization|
|Session 2: Feature prioritization (n=22)||Stage 2: prioritization (Kano) and ideation on potential outcomes||Consultants or partnership: participant feedback via Kano and decisions on prioritization||Mural, Zoom, and REDCapa||Paper survey, butcher paper, sticky notes, markers, and bullseye visualization|
|Session 3: Iterative prototyping 3.1: Wireframing and prototyping (n=4)3.2: Account creation (n=4)3.3: Aesthetics(n=1)||Stage 3: prototyping||Partnership: participants decide how the resource displays each feature||Mural, Zoom, and Justinmind||Butcher paper, sticky notes, markers, PowerPoint to create first draft screen visuals, prototyping software|
|Session 4: Usability testing (n=2)||Heuristic evaluation: cognitive walkthrough||N/Ab||Zoom, REDCap, and the TGHIRc||The TGHIR, paper, and pencil|
aREDCap: Research Electronic Data Capture.
bN/A: not applicable.
cTGHIR: Transgender Health Information Resource.
Design Session 1: Exploration of Potential Features
We described and characterized engagement in this session and provided findings in the results. Overall, design session 1 included 4 participants who identified as transgender (2/4, 50%) and nonbinary (2/4, 50%). The races reported by participants included American Indian (1/4, 25%), multiple races (1/4, 25%), and White (2/4, 50%). Age ranges were as follows: 20 to 29 years (1/4, 25%) and 30 to 39 years (3/4, 75%). Participants lived in rural areas (1/4, 25%) and urban locations (3/4, 75%).
Design session 1 was exploratory, informed by stage 1 of participatory design approach by Spinuzzi  and facilitated by the first author, a member of the research team. The focus was on gathering information from participants about the mobile resources end users in our target audience liked using and what features made their experience with the resource enjoyable. Participants developed personas through an exercise in which they ascribed feelings, values, and behaviors, resulting in potential use cases. An example of a persona statement is provided in .
Mural (), a web-based collaborative platform, allowed us to use the electronic counterparts of butcher paper, sticky notes, electronic pens with assorted color ink, and distinct colors and sizes of fonts.
We held a 10-minute Mural training session to optimize the time we had with the participants. Owing to the exploratory nature of design session 1, multiple brainstorming activities were conducted. Typically, participants wrote on sticky notes and organized their ideas on a whiteboard. Mural allowed for this functionality. In this session, we asked participants a series of questions about their mobile resource preferences, including, (1) What mobile apps do you like and why, (2) What mobile apps do you dislike and why, (3) What mobile apps do you use often and what makes them reusable, (4) What mobile apps have you stopped using and why, (5) What makes information on a mobile app credible, and (6) What websites do you use to find health information? Design session 1 also included an activity in which participants placed mobile resource features identified in the brainstorming activity on a bullseye image indicating how important the feature was to them.
Design Session 2: Feature Prioritization
Design session 2 focused on discovering and prioritizing resource features identified in the first design session using stage 2, the discovery processes, of the participatory design approach . The prioritization of features was organized using a Kano model of customer satisfaction [ - ]. This method assessed participants’ opinions regarding a feature being implemented [ ]. The transition to web-based implementation required our design team to collect the Kano survey of customer satisfaction through REDCap instead of on paper and in-person. In this exercise, we used Mural to display the features generated by participants in design session 1 so the feature could be prioritized. The participants went through the survey as a group but responded individually.
After the Kano survey was completed, a second round of placing features and categories on a priority bullseye visualization () was conducted as a design exercise, furthering the discussion on what features were the most desirable to end users and should therefore be developed first. The design session 2 bullseye exercise was compared with how features were prioritized using the Kano survey. In addition, this exercise allowed participants to group the features into categories, such as in-person card sort, by grouping different sticky notes that referred to similar or the same type of features [ , ]. The last activity in design session 2 was the initial prototyping of the home menu and health resource preview, which was performed in Mural.
Design Session 3: Iterative Prototyping
Design session 3.1 and design session 3.2 focused on wireframing, a process in which a sketch is made of what a product, in this case, a user interface, may look like. The wireframe was used as a starting point for the design work  and prototyping of the health information resource as an intentional and planned health information–seeking experience in stage 3 of the participatory design approach [ ]. This work was performed using the Justinmind tool, a program that allows designers to rapidly create interfaces and modify them in real time. Using Justinmind, we created wireframes and asked participants for input on the design and implied functionality using Zoom. Participants first responded to the design elements of the baseline design created by the researchers. In traditional in-person sessions, the pencil and paper design methodology allowed for a quick iteration of the initial design during an in-person session. In this web-based design session, the use of a baseline design allowed the researcher to engage more quickly and in greater detail about the interface because time was not used to set up the initial design. Design session 3.3 was held with one participant to collect insights into aesthetics, or look and feel, of the TGHIR.
Design Session 4: Usability Goal-Oriented
The fourth and final design session occurred after the TGHIR was built and consisted of a cognitive walkthrough heuristic evaluation, in which an experienced mobile health researcher and a behavioral scientist were asked to perform navigationally based tasks, assuming that most tasks embedded in the TGHIR would be goal-orientated , for example, find and like a resource. Participants installed the resource on their phones for an authentic experience.
In the usability testing sessions, Zoom, REDCap, and TGHIR were used. In our initial plans, this interaction would have been in-person and the participant would have had the TGHIR in their hands, as we observed their use of the resource and recorded the necessary usability data. Instead, design team members observed the participants through a Zoom connection. Usability evaluation tasks () were described and completed by the participants [ , ]. These tasks were prioritized because they were associated with the features participants identified as important for the TGHIR to access credible health information.
|Task||Action to be completed|
|1||Create account and view consent and privacy language|
|2||Select preferred health information categories|
|3||Find a specific health information item using the category cards|
|4||Find and use the filter to narrow the resources to a relevant informational item|
|5||Like a resource|
|6||Bookmark a resource|
|7||Locate and use the search function to find a specific health information resource|
|8||Send a message to the developers|
|9||Share a new resource for the community with the developers|
|10||Find the most liked health information resource|
We analyzed the video recording from design session 1 and the artifact created in the Mural collaborative space to describe and characterize web-based engagement. Design session 1 was recorded using the Zoom tool and transcribed by a professional transcriptionist. A research team member performed rapid analysis  of the transcriptions to quickly identify key points and comments that reflected participant engagement and major design decisions. Exemplar quotes were provided to highlight the impact of comments on the final design and the resulting features developed in the resource. Quantitative data included length of each design session exercise, number of verbal comments by all participants, total verbal comments, number of times the Zoom camera was turned off by participants, number of Mural interactions (creation or manipulation of digital Mural content) by participants, and total Mural interactions by design exercise. For a verbal comment or Mural interaction to be counted, the comment or interaction had to be considered influential in advancing the design process. Comments or interactions that expressed confirmation or agreement were not considered influential.
Informed verbal consent was obtained from all participants. Each participant was paid US $50 for each of the 1.5-hour design sessions they attended. Project approval was obtained from the University of Colorado’s Human Research Ethics Committee, Colorado Multiple Institutional Board (COMIRB# 19-1562).
Eligibility screening was performed using Zoom. A total of 41 individuals were screened for eligibility; of these, 27 (66%) were eligible and available to participate in at least one of the design sessions. Mural relied on well-established computing conventions to add, edit, and delete content and was, therefore, familiar to the participants who quickly mastered the skills to work alongside the research staff in the collaborative space.
Design Session 1
In design session 1, participants identified the following specific health information for inclusion in the resource: affirming care (supportive care for the TGD community ), affordable medical options, information on transition, information on successful transitions (transitions were associated with the period during which a person began to live according to their gender identity rather than the gender they were thought to be at birth [ ]), family resources, community support, and medically verified information.
Responses to the questions addressing participants’ mobile app preferences are presented in.
We provided the number of verbal comments and web-based interactions in design session 1 to show the volume of participant engagement in. Design session 1 was 119-minutes long; there was an average of 1.87 web-based points of engagement every minute. The researchers turned off their cameras to focus on the participants. Two participants experienced bandwidth trouble during the first exercise and chose to turn off their cameras to contribute to the web-based participatory design process. These 2 participants did not provide any other reason for their cameras being turned off and both continued to contribute despite the bandwidth problems they experienced.
Additional qualitative evidence for the effectiveness of the web-based participatory design process is presented in. Participant insights from the first design session remained prominent throughout the design process and directly affected the course of the design work. Participant’s verbal interactions were evident in the final TGHIR features.
|What mobile apps do you like and why?|
|What mobile apps do you dislike and why?|
|What mobile apps do you use often and what makes them reusable?|
|Reasons for not using an app|
|What makes information on a mobile app credible?|
|What websites do you use to find health information?|
aUI: user interface.
bUX: user experience.
|Mural exercise||Exercise time (length of discussion)||Verbal comments by participants, n||Verbal comments per exercise, n||Web-based interactions by participants, n||Web-based interactions per exercise, n|
|What apps do you like||6:10-14:50||2||2||0||1||5||8||10||3||7||28|
|What apps do you use||14:51-22:29||1||1||3||3||8||11||10||7||10||38|
|Value proposition generation||1:34:05-1:55:10||2||1||0||0||3||4||4||2||4||14|
|Total||1:58:55 (including breaks)||29||28||9||13||79||63||67||47||45||222|
bTGD: transgender and gender-diverse.
|Comment||Impact on design||Influence on design|
|1.3||“I think having a channel in which folks can reach out if there’s an issue with information on the app...message customer support, call customer support or some kind of way that it’s not just, you know, you’re not searching really, you know, a long way through the app or trying to find a human to connect with within even an issue or a question”||Customer support as a priority within the app—visible, easily accessible. Important to make users feel valued and heard.||Contact Us page with the option of sending different types of messages to the TGHIRa development team.|
|1.5||“I think it would be good if there is a place for it [the resource] to remember your name but also not to make it too cumbersome should you decide to change it. If it remembers your names also having an option for pronouns, even if it doesn’t pop up anywhere, it’s nice to have that affirmation.”||Function to remember names and customize pronouns within the resource. Demonstrates understanding of target audience.||Providing pronouns was eventually scrapped to avoid collecting data the resource would not use. This is the settings menu based on the comment above. The design attempts to make changing settings easy.|
|1.5||“My Verizon [app] can update some things on your account, but otherwise it’s just a giant ad for you to upgrade your system. I think interactivity, there’s a reason why you’ve downloaded an app instead of going to the website.”||Information should be presented in a mobile app that fosters easy access with an intuitive interactive navigation so the app could be used at any time and in any place.||The TGHIR was design and developed for iOS and Android, the 2 most common mobile app operating systems.|
|1.6||“I have the [Name], My Chart open. And one of the options is you can personalize it. The personalization is just changing the color for the frame of the app. That seems pointless to me...yeah, it just seems like such a superfluous option. I would rather move the icons around on the homepage...there might be something that I like to check more than other people. Moving the menu around would be a nice personalization. But changing the color is not worth it.”||If the resource is going to include customization options, it should go beyond the ability to simply change the colors on the interface, eg, the option to move icons around based on relevancy to the individual user. The orange cards are categories of interest selected by the user.||The Health Resource Cards could be pinned to the top of the page making them easier to return to during repeated use. Once the resource was pinned, the TGHIR app changes the card from blue to orange.|
aTGHIR: Transgender Health Information Resource.
We found that using a web-based suite of collaborative tools with participants, we were able to effectively engage in productive discussions and make design decisions for the development of TGHIR. This approach to design work, in our case, was seamless and did not limit participants from engaging and providing meaningful influence on the resulting health information resources. The value of web-based recruitment and design sessions should be underscored because of the relative social safety of TGD participants. This recruitment method may be a strength for engagement and have a positive impact on TGD involvement in research by creating safe web-based spaces for TGD involvement.
We observed consistent engagement throughout design session 1. Upon reflection and analysis of the data, we found that the immediacy of engagement was impressive. It did not take participants much time to get involved and discuss design issues with their fellow participants. The dual interaction of manipulating web-based content in the Mural document and discussing the topic at hand may have led to a more immediate collaboration.
This study shed light on a web-based methodological approach to co-design health information resources within TGD communities. Through their involvement and enthusiasm for the work, participants indicated that a web-based approach to design was appropriate and can be used instead of resource-intensive in-person gatherings. Queer and trans communities have embraced digital technologies in radically affirming ways [, ] to move beyond the acceptance of unsatisfactory options. The uptake of web-based design, and the necessary digital technologies, in TGD communities is feasible.
Mural interactions indicated that participants were consistently present throughout the design process. With 222 Mural interactions across 4 participants, it was evident that these individuals were highly engaged on the process. Whether this level of engagement can be replicated or generalized to other communities and topics requires further research. The verbal comments provided evidence that the interactions were process-oriented and moved the discussion to inform the development of TGHIR. Furthermore, specific design decisions were made owing to the input of the participants, providing evidence of their importance in the web-based design process and speaking to the effectiveness of web-based participatory design.
Strengths, Limitations, and Future Directions
The strength of this project was the introduction of a method to describe and characterize engagement and interactions during participatory design facilitated by web-based means. The first limitation was that this study evaluated the engagement of participants from the TGD community but did not empirically compare the quality or quantity of engagement with an in-person design process. Future research should examine in-person design sessions compared with web-based design sessions to evaluate and compare the quality and quantity of engagement and interactions. Second, our recruitment method used Facebook to identify participants. This approach yielded a predominantly educated, professional, urban, and white sample. Although we recruited enough TGD individuals for the design of the TGHIR, we acknowledge that this sample may not have been representative of the overall population. As a result, we sought and were awarded funding to test the resources using a diverse research sample. This study is currently underway. Third, the small number of participants was a limitation. In future work, more participants should be involved in the participatory design process to determine whether high levels of web-based participation are maintained in larger samples. Finally, although web-based means of participation helped during the COVID-19 pandemic, a limitation was that it might have also been difficult to reach individuals who did not have the means or ability to access the requisite resources for web-based collaboration, such as high-speed internet and private space . It will be important to keep this accessibility issue in mind when recruiting participants.
Our results had important implications for the use of web-based methodologies in the design of health information resources. Web-based participatory design can support opportunities to contribute despite the potential logistical barriers of in-person design sessions by offering multiple convenient design session times and multiple interaction options. In addition, this approach is helpful when recruiting members from marginalized communities that are small and geographically dispersed, especially rural communities. Not only does the web-based methodological approach work during a pandemic but it may also help when there is historic distrust of research and health care from a community that has been stigmatized and experienced discrimination, such as the TGD community, by researchers and medical providers.
Our evaluation of the web-based participatory design indicated that web-based design sessions can engage participants in creating satisfactory interfaces for accessing and consuming health and medical information. Obtaining web-based input from participants was possible and efficient. Web-based recruitment is also possible for individuals who belong to marginalized communities and provides a platform through which these individuals can safely communicate with others in their community to design health information resources. Integrating web-based platforms can effectively engage participants and yield a positive user experience. Multiple participants reported that a health information resource of this nature would have been helpful in their journey toward gender identity exploration or gender transition.
We would like to acknowledge all members and supporters of the gender-diverse community, especially our participatory co-designers and our Community Advisory Board members. The content of this paper is solely the responsibility of the authors and does not necessarily represent the official views of the National Library of Medicine. This work was supported by funding from the National Institutes of Health, National Library of Medicine, grant GO8 LM013200-01, project name: A Health Information Resource Technology to Reduce Disparities in Transgender Health.
Relevant data associated with, created, and used for the evaluation presented in this paper are openly available from the University of Colorado Strauss Health Sciences Library, Anschutz Medical Campus, Data Repository .
BM was responsible for designing and facilitating the participatory design process, coding the qualitative manuscript and web-based interactions, and served as the lead author. AS was responsible for designing the methodology, agile development of the resource, and served as the second author. KY was responsible for obtaining data in usability tests, coding the qualitative transcript and web-based interactions, and reviewing and editing the manuscript. KD was responsible for cataloging the content in the resource, designing the search for credible transgender health resources, and for reviewing and editing the manuscript. MA was responsible for the design of the methodology, the interpretation of data, and for reviewing and editing the manuscript. BDH was responsible for designing the qualitative methodology for focus groups, interpretation of focus group data, and reviewing and editing the manuscript. RSL was responsible for designing the methodology, interpretation of the data, and for reviewing and editing the manuscript. BMK was responsible for designing the methodology, interpretation of the data, and for reviewing and editing the manuscript. LMS was responsible for designing the methodology, interpretation of the data, for reviewing and editing the manuscript, and served as the senior author.
Conflicts of Interest
- Walch A, Davidge-Pitts C, Safer JD, Lopez X, Tangpricha V, Iwamoto SJ. Proper care of transgender and gender diverse persons in the setting of proposed discrimination: a policy perspective. J Clin Endocrinol Metab 2021 Jan 23;106(2):305-308 [FREE Full text] [CrossRef] [Medline]
- Hswen Y, Zhang A, Sewalk KC, Tuli G, Brownstein JS, Hawkins JB. Investigation of geographic and macrolevel variations in LGBTQ patient experiences: longitudinal social media analysis. J Med Internet Res 2020 Jul 31;22(7):e17087 [FREE Full text] [CrossRef] [Medline]
- Sun CJ, Anderson KM, Kuhn T, Mayer L, Klein CH. A sexual health promotion app for transgender women (trans women connected): development and usability study. JMIR Mhealth Uhealth 2020 May 12;8(5):e15888 [FREE Full text] [CrossRef] [Medline]
- Brown GR, Jones KT. Mental health and medical health disparities in 5135 transgender veterans receiving healthcare in the veterans health administration: a case-control study. LGBT Health 2016 Apr;3(2):122-131. [CrossRef] [Medline]
- Heng A, Heal C, Banks J, Preston R. Transgender peoples’ experiences and perspectives about general healthcare: a systematic review. Int J Transgend 2018 Nov 22;19(4):359-378. [CrossRef]
- Baldwin A, Dodge B, Schick VR, Light B, Scharrs PW, Herbenick D, et al. Transgender and genderqueer individuals' experiences with health care providers: what's working, what's not, and where do we go from here? J Health Care Poor Underserved 2018;29(4):1300-1318. [CrossRef] [Medline]
- Bartholomaeus C, Riggs DW, Sansfaçon AP. Expanding and improving trans affirming care in Australia: experiences with healthcare professionals among transgender young people and their parents. Health Sociol Rev 2021 Mar;30(1):58-71. [CrossRef] [Medline]
- Ettner R, Ettner F, White T. Choosing a surgeon: an exploratory study of factors influencing selection of a gender affirmation surgeon. Transgend Health 2016 Jul 1;1(1):124-128 [FREE Full text] [CrossRef] [Medline]
- Chan B, Skocylas R, Safer JD. Gaps in transgender medicine content identified among Canadian medical school curricula. Transgend Health 2016 Jul 1;1(1):142-150 [FREE Full text] [CrossRef] [Medline]
- Johnston CD, Shearer LS. Internal medicine resident attitudes, prior education, comfort, and knowledge regarding delivering comprehensive primary care to transgender patients. Transgend Health 2017 Jul 1;2(1):91-95 [FREE Full text] [CrossRef] [Medline]
- Bakko M, Kattari SK. Transgender-related insurance denials as barriers to transgender healthcare: differences in experience by insurance type. J Gen Intern Med 2020 Jun;35(6):1693-1700 [FREE Full text] [CrossRef] [Medline]
- Learmonth C, Viloria R, Lambert C, Goldhammer H, Keuroghlian AS. Barriers to insurance coverage for transgender patients. Am J Obstet Gynecol 2018 Sep;219(3):272.e1-272.e4. [CrossRef] [Medline]
- Carosella EA, Huerta L, Galea JT, Lecca L, Ramos K, Hernandez G, et al. Towards increasing equity in healthcare among transgender women: a cross-sectional analysis of self-reported needs and health service utilization in Lima, Peru. medRxiv. Preprint posted on online on Feb 2, 2022. [CrossRef]
- Chisolm-Straker M, Jardine L, Bennouna C, Morency-Brassard N, Coy L, Egemba MO, et al. Transgender and gender nonconforming in emergency departments: a qualitative report of patient experiences. Transgend Health 2017 Feb 1;2(1):8-16 [FREE Full text] [CrossRef] [Medline]
- Bauer GR, Hammond R, Travers R, Kaay M, Hohenadel KM, Boyce M. "I don't think this is theoretical; this is our lives": how erasure impacts health care for transgender people. J Assoc Nurses AIDS Care 2009;20(5):348-361. [CrossRef] [Medline]
- Hatzenbuehler ML, Pachankis JE. Stigma and minority stress as social determinants of health among lesbian, gay, bisexual, and transgender youth: research evidence and clinical implications. Pediatr Clin North Am 2016 Dec;63(6):985-997. [CrossRef] [Medline]
- Mizock L, Mueser KT. Employment, mental health, internalized stigma, and coping with transphobia among transgender individuals. Psychol Sex Orientat Gend Divers 2014;1(2):146-158. [CrossRef]
- Lombardi EL, Wilchins RA, Priesing D, Malouf D. Gender violence: transgender experiences with violence and discrimination. J Homosex 2001;42(1):89-101. [CrossRef] [Medline]
- Testa RJ, Sciacca LM, Wang F, Hendricks ML, Goldblum P, Bradford J, et al. Effects of violence on transgender people. Prof Psychol Res Pract 2012 Oct;43(5):452-459. [CrossRef]
- Augustaitis L, Merrill LA, Gamarel KE, Haimson OL. Online transgender health information seeking: facilitators, barriers, and future directions. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 2021 May Presented at: CHI '21; May 8-13, 2021; Yokohama, Japan p. 1-14. [CrossRef]
- Wirtz AL, Poteat T, Radix A, Althoff KN, Cannon CM, Wawrzyniak AJ, American Cohort To Study HIV Acquisition Among Transgender Women (LITE). American cohort to study HIV acquisition among transgender women in high-risk areas (the LITE study): protocol for a multisite prospective cohort study in the Eastern and Southern United States. JMIR Res Protoc 2019 Oct 03;8(10):e14704 [FREE Full text] [CrossRef] [Medline]
- Kosenko K, Rintamaki L, Raney S, Maness K. Transgender patient perceptions of stigma in health care contexts. Med Care 2013 Sep;51(9):819-822. [CrossRef] [Medline]
- Flores AR, Brown TN, Herman JL. Race and ethnicity of adults who identify as transgender in the United States. The Williams Institute, UCLA. 2016 Oct. URL: https://williamsinstitute.law.ucla.edu/wp-content/uploads/Race-Ethnicity-Trans-Adults-US-Oct-2016.pdf [accessed 2022-03-07]
- Pho AT, Bakken S, Lunn MR, Lubensky ME, Flentje A, Dastur Z, et al. Online health information seeking, health literacy, and human papillomavirus vaccination among transgender and gender-diverse people. J Am Med Inform Assoc 2022 Jan 12;29(2):285-295 [FREE Full text] [CrossRef] [Medline]
- Hardy J, Vargas S. Participatory design and the future of rural LGBTQ communities. In: Companion Publication of the 2019 on Designing Interactive Systems Conference 2019 Companion. 2019 Jun Presented at: DIS '19 Companion; June 23-28, 2019; San Diego, CA, USA p. 195-199. [CrossRef]
- Evans YN, Gridley SJ, Crouch J, Wang A, Moreno MA, Ahrens K, et al. Understanding online resource use by transgender youth and caregivers: a qualitative study. Transgend Health 2017 Aug 1;2(1):129-139 [FREE Full text] [CrossRef] [Medline]
- Selkie E, Adkins V, Masters E, Bajpai A, Shumer D. Transgender adolescents' uses of social media for social support. J Adolesc Health 2020 Mar;66(3):275-280. [CrossRef] [Medline]
- The Trevor Project | For Young LGBTQ Lives. The Trevor Project. URL: https://www.thetrevorproject.org/ [accessed 2021-11-18]
- MedlinePlus - Health Information from the National Library of Medicine. URL: https://medlineplus.gov/ [accessed 2021-11-18]
- Magee JC, Bigelow L, Dehaan S, Mustanski BS. Sexual health information seeking online: a mixed-methods study among lesbian, gay, bisexual, and transgender young people. Health Educ Behav 2012 Jun;39(3):276-289. [CrossRef] [Medline]
- Hawkins B, Gieseking JJ. Seeking ways to our transgender bodies, by ourselves: rationalizing transgender‐specific health information behaviors. Proc Assoc Info Sci Tech 2017 Oct 24;54(1):702-704. [CrossRef]
- MacDonell KK, Wang B, Phanuphak N, Janamnuaysook R, Srimanus P, Rongkavilit C, et al. Optimizing an mHealth intervention to improve uptake and adherence to HIV pre-exposure prophylaxis in young transgender women: protocol for a multi-phase trial. JMIR Res Protoc 2022 May 19;11(5):e37659 [FREE Full text] [CrossRef] [Medline]
- Mobile Fact Sheet: Mobile phone ownership over time. Pew Research Center. 2021 Apr 7. URL: https://www.pewresearch.org/internet/fact-sheet/mobile/ [accessed 2021-05-13]
- Gurstein M. Effective use: a community informatics strategy beyond the Digital Divide. First Monday 2007 Oct 27;8(12). [CrossRef]
- Hwang A, Truong KN, Mihailidis A. Using participatory design to determine the needs of informal caregivers for smart home user interfaces. In: Proceedings of the 6th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) and Workshops. 2012 Presented at: ICST '12; May 21-24, 2012; San Diego, CA, USA p. 41-48. [CrossRef]
- Lindsay S, Jackson D, Schofield G, Olivier P. Engaging older people using participatory design. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2012 Presented at: CHI '12; May 5-10, 2012; Austin, TX, USA p. 1199-1208. [CrossRef]
- Kyakulumbye S, Pather S. Understanding ICT adoption amongst SMEs in Uganda: towards a participatory design model to enhance technology diffusion. Afr J Sci Technol Innov Dev 2022;14(1):49-60. [CrossRef]
- Robertson T, Simonsen J. Participatory design: an introduction. In: Simonsen J, Robertson J, editors. Routledge International Handbook of Participatory Design. London, UK: Routledge; 2013.
- Wilkinson CR, De Angeli A. Applying user centred and participatory design approaches to commercial product development. Des Stud 2014 Nov;35(6):614-631. [CrossRef]
- Siemens L. The balance between on-line and in-person interactions: methods for the development of digital humanities collaboration. Digital Stud 2011 May 17;2(1). [CrossRef]
- Harrington C, Erete S, Piper AM. Deconstructing community-based collaborative design: towards more equitable participatory design engagements. Proc ACM Hum Comput Interact 2019 Nov 07;3(CSCW):1-25. [CrossRef]
- Cherry C, Macredie RD. The importance of context in information system design: an assessment of participatory design. Requir Eng 1999 Jul 1;4(2):103-114. [CrossRef]
- Salvo MJ. Ethics of engagement: user-centered design and rhetorical methodology. Tech Commun Q 2001 Jul;10(3):273-290. [CrossRef]
- Bossen C, Dindler C, Iversen OS. Evaluation in participatory design: a literature survey. In: Proceedings of the 14th Participatory Design Conference: Full papers - Volume 1. 2016 Aug Presented at: PDC '16; August 15-19, 2016; Aarhus, Denmark p. 151-160. [CrossRef]
- Drain A, Shekar A, Grigg N. Insights, solutions and empowerment: a framework for evaluating participatory design. CoDesign 2018 Nov 01;17(1):1-21. [CrossRef]
- Thoresen K. Prototyping organizational change. In: Proceedings of the 1st Biennial Participatory Design Conference. 1990 Jan 1 Presented at: PDC '90; March 31-April 1, 1990; Seattle, WA, USA. [CrossRef]
- Bossen C, Dindler C, Iversen OS. User gains and PD aims: assessment from a participatory design project. In: Proceedings of the 11th Biennial Participatory Design Conference. 2010 Presented at: PDC '10; November 29-December 3, 2010; Sydney, Australia p. 141-150. [CrossRef]
- Ravn Jakobsen P, Hermann AP, Søndergaard J, Wiil UK, Clemensen J. Development of an mHealth application for women newly diagnosed with osteoporosis without preceding fractures: a participatory design approach. Int J Environ Res Public Health 2018 Feb 13;15(2):330 [FREE Full text] [CrossRef] [Medline]
- Norman CD, Haresign H, Mehling C, Bloomberg H. Exploring the feasibility and potential of virtual panels for soliciting feedback on nutrition education materials: a proof-of-concept study. JMIR Public Health Surveill 2016 Apr 19;2(1):e18 [FREE Full text] [CrossRef] [Medline]
- Anderson J, Ashraf N, Douther C, Jack MA. Presence and usability in shared space virtual conferencing: a participatory design study. Cyberpsychol Behav 2001 Apr;4(2):287-305. [CrossRef] [Medline]
- Arnold J, Barrow A, Teynor R. Codesign in virtual space: including everyday people in product design. In: Proceedings of the 2007 National Education Symposium of the Industrial Designers Society of America. 2007 Presented at: IDSA '07; 2007; San Francisco, CA, USA URL: https://www.idsa.org/sites/default/files/Arnold_Barrow_Teynor-CoDesign_Virtual_Space.pdf
- Farshchian BA, Divitini M. Using email and WWW in a distributed participatory design project. SIGGROUP Bull 1999 Apr;20(1):10-15. [CrossRef]
- Hess J, Pipek V. Community-driven development: approaching participatory design in the online world. Design Issues 2012 Jul;28(3):62-76. [CrossRef]
- LaMonica HM, Davenport TA, Roberts AE, Hickie IB. Understanding technology preferences and requirements for health information technologies designed to improve and maintain the mental health and well-being of older adults: participatory design study. JMIR Aging 2021 Jan 06;4(1):e21461 [FREE Full text] [CrossRef] [Medline]
- Braa J, Sahay S. Health information systems program: participatory design within the HISP network. In: Simonsen J, Robertson T, editors. Routledge International Handbook of Participatory Design. London, UK: Routledge; 2013.
- Martin-Hammond A, Vemireddy S, Rao K. Exploring older adults' beliefs about the use of intelligent assistants for consumer health information management: a participatory design study. JMIR Aging 2019 Dec 11;2(2):e15381 [FREE Full text] [CrossRef] [Medline]
- Byrne E, Sahay S. Participatory design for social development: a South African case study on community-based health information systems. Inf Technol Dev 2007;13(1):71-94. [CrossRef]
- Allin S, Shepherd J, Tomasone J, Munce S, Linassi G, Hossain SN, et al. Participatory design of an online self-management tool for users with spinal cord injury: qualitative study. JMIR Rehabil Assist Technol 2018 Mar 21;5(1):e6 [FREE Full text] [CrossRef] [Medline]
- Bilodeau K, Gouin MM, Lecours A, Lederer V, Durand MJ, Kilpatrick K, et al. Acceptability and feasibility of a return-to-work intervention for posttreatment breast cancer survivors: protocol for a co-design and development study. JMIR Res Protoc 2022 Apr 22;11(4):e37009 [FREE Full text] [CrossRef] [Medline]
- Abrahamsson P, Salo O, Ronkainen J, Warsta J. Agile software development methods: review and analysis. arXiv. Preprint posted online on September 9, 2017. [CrossRef]
- Funk A, Van Borek N, Taylor D, Grewal P, Tzemis D, Buxton JA. Climbing the "ladder of participation": engaging experiential youth in a participatory research project. Can J Public Health 2012 May 24;103(4):e288-e292 [FREE Full text] [CrossRef] [Medline]
- Arnstein's Ladder of Citizen Participation. The Citizen's Handbook. URL: https://www.citizenshandbook.org/arnsteinsladder.html [accessed 2022-06-02]
- One Colorado - LGBTQ Advocacy Organization. One Colorado. URL: https://one-colorado.org/ [accessed 2021-09-27]
- Video Conferencing, Cloud Phone, Webinars, Chat, Virtual Events. Zoom. URL: https://zoom.us/ [accessed 2021-06-09]
- Spinuzzi C. The methodology of participatory design. Tech Commun 2005 May 1;52(2):163-174.
- MURAL is a digital workspace for visual collaboration. MURAL. URL: https://www.mural.co/ [accessed 2021-07-09]
- REDCap. URL: https://www.project-redcap.org/ [accessed 2021-07-09]
- Free prototyping tool for web and mobile apps. Justinmind. URL: https://www.justinmind.com/ [accessed 2021-07-09]
- Sauerwein E, Bailom F, Matzler K, Hinterhuber HH. The Kano model: how to delight your customers. In: Proceedings of the 9th International Working Seminar on Production Economics. 1996 Jan 1 Presented at: IWSPE '96; February 19-23, 1996; Innsbruck, Austria p. 313-327. [CrossRef]
- Chen CC, Chuang MC. Integrating the Kano model into a robust design approach to enhance customer satisfaction with product design. Int J Prod Econ 2008 Aug;114(2):667-681. [CrossRef]
- Mikulić J, Prebežac D. A critical review of techniques for classifying quality attributes in the Kano model. Manag Serv Qual 2011 Jan 25;21(1):46-66. [CrossRef]
- Xu Q, Jiao RJ, Yang X, Helander M, Khalid HM, Opperud A. An analytical Kano model for customer need analysis. Des Stud 2009 Jan;30(1):87-110. [CrossRef]
- Spencer D, Warfel T. Card Sorting: A Definitive Guide. Boxes and Arrows. 2004 Apr 7. URL: https://boxesandarrows.com/card-sorting-a-definitive-guide/ [accessed 2021-12-22]
- Paul CL. A modified Delphi approach to a new card sorting methodology. J Usability Stud 2008 Nov;4(1):7-30.
- Hamm MJ. Wireframing Essentials. Birmingham, UK: Packt Publishing; 2014.
- Blackmon MH, Polson PG, Kitajima M, Lewis C. Cognitive walkthrough for the web. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2002 Apr Presented at: CHI '02; April 20-25, 2002; Minneapolis, MN, USA p. 463-470. [CrossRef]
- Monkman H, Kushniruk A. A health literacy and usability heuristic evaluation of a mobile consumer health application. Stud Health Technol Inform 2013;192:724-728. [Medline]
- Khajouei R, Zahiri Esfahani M, Jahani Y. Comparison of heuristic and cognitive walkthrough usability evaluation methods for evaluating health information systems. J Am Med Inform Assoc 2017 Apr 01;24(e1):e55-e60 [FREE Full text] [CrossRef] [Medline]
- Gale RC, Wu J, Erhardt T, Bounthavong M, Reardon CM, Damschroder LJ, et al. Comparison of rapid vs in-depth qualitative analytic methods from a process evaluation of academic detailing in the Veterans Health Administration. Implement Sci 2019 Feb 01;14(1):11 [FREE Full text] [CrossRef] [Medline]
- Frequently Asked Questions about Transgender People. National Center for Transgender Equality. 2016 Jul 9. URL: https://transequality.org/issues/resources/frequently-asked-questions-about-transgender-people [accessed 2022-08-15]
- Floegel D, Costello KL. Entertainment media and the information practices of queer individuals. Libr Inf Sci Res 2019 Jan;41(1):31-38. [CrossRef]
- Wagner TL, Kitzie VL, Lookingbill V. Transgender and nonbinary individuals and ICT-driven information practices in response to transexclusionary healthcare systems: a qualitative study. J Am Med Inform Assoc 2022 Jan 12;29(2):239-248 [FREE Full text] [CrossRef] [Medline]
- Hamnvik OP, Agarwal S, AhnAllen CG, Goldman AL, Reisner SL. Telemedicine and inequities in health care access: the example of transgender health. Transgend Health 2022 Apr 01;7(2):113-116. [CrossRef]
- Morse B, Soares A, Ytell K, DeSanto K, Allen M, Holliman BD, et al. TGHIR design session 1 coding. University of Colorado Strauss Health Sciences Library, Anschutz Medical Campus, Data Repository. 2022. URL: https://digitalcollections.cuanschutz.edu/work/ns/3e10312c-3950-4a49-9b43-a5fadb4349c5 [accessed 2022-12-09]
|LGBTQ: lesbian, gay, bisexual, transgender, and queer|
|REDCap: Research Electronic Data Capture|
|TGD: transgender and gender-diverse|
|TGHIR: Transgender Health Information Resource|
Edited by S Woods; submitted 17.03.22; peer-reviewed by S Wei, M Paton, T Wagner; comments to author 10.05.22; revised version received 17.06.22; accepted 15.11.22; published 10.01.23Copyright
©Brad Morse, Andrey Soares, Kate Ytell, Kristen DeSanto, Marvyn Allen, Brooke Dorsey Holliman, Rita S Lee, Bethany M Kwan, Lisa M Schilling. Originally published in Journal of Participatory Medicine (https://jopm.jmir.org), 10.01.2023.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in Journal of Participatory Medicine, is properly cited. The complete bibliographic information, a link to the original publication on https://jopm.jmir.org, as well as this copyright and license information must be included.