Participatory Methods to Engage Health Service Users in the Development of Electronic Health Resources: Systematic Review

Background When health service providers (HSP) plan to develop electronic health (eHealth) resources for health service users (HSU), the latter’s involvement is essential. Typically, however, HSP, HSU, and technology developers engaged to produce the resources lack expertise in participatory design methodologies suited to the eHealth context. Furthermore, it can be difficult to identify an established method to use, or determine how to work stepwise through any particular process. Objective We sought to summarize the evidence about participatory methods and frameworks used to engage HSU in the development of eHealth resources from the beginning of the design process. Methods We searched for studies reporting participatory processes in initial development of eHealth resources from 2006 to 2016 in 9 bibliographic databases: MEDLINE, EMBASE, CINAHL, PsycINFO, Emcare, Cochrane Library, Web of Science, ACM Guide to Computing Literature, and IEEE Xplore. From 15,117 records initially screened on title and abstract for relevance to eHealth and early participatory design, 603 studies were assessed for eligibility on full text. The remaining 90 studies were rated by 2 reviewers using the Mixed Methods Appraisal Tool Version 2011 (Pluye et al; MMAT) and analyzed with respect to health area, purpose, technology type, and country of study. The 30 studies scoring 90% or higher on MMAT were included in a detailed qualitative synthesis. Results Of the 90 MMAT-rated studies, the highest reported (1) health areas were cancer and mental disorders, (2) eHealth technologies were websites and mobile apps, (3) targeted populations were youth and women, and (4) countries of study were the United States, the United Kingdom, and the Netherlands. Of the top 30 studies the highest reported participatory frameworks were User-Centered Design, Participatory Action Research Framework, and the Center for eHealth Research and Disease Management (CeHRes) Roadmap, and the highest reported model underpinning development and engagement was Social Cognitive Theory. Of the 30 studies, 4 reported on all the 5 stages of the CeHRes Roadmap. Conclusions The top 30 studies yielded 24 participatory frameworks. Many studies referred to using participatory design methods without reference to a framework. The application of a structured framework such as the CeHRes Roadmap and a model such as Social Cognitive Theory creates a foundation for a well-designed eHealth initiative that ensures clarity and enables replication across participatory design projects. The framework and model need to be clearly articulated and address issues that include resource availability, responsiveness to change, and the criteria for good practice. This review creates an information resource for future eHealth developers, to guide the design of their eHealth resource with a framework that can support further evaluation and development. Trial Registration PROSPERO CRD42017053838; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=53838


Rationale
Individuals are increasingly being offered access to health services via electronic health (eHealth), sometimes called digital health, that is, health-related electronic resources that connect them with health service providers (HSP) over the internet. Examples include websites, portals, social media sites, serious games, mobile apps, wearable self-monitoring devices, online learning sites, telehealth platforms, and shareable electronic health records. Patients, clients, or consumers in this review are called health service users (HSU). They may require services to support their physical health, mental health, and well-being in the broadest sense of the World Health Organization's definition (Table 1) [1].
The involvement of HSU as full participants in eHealth innovations responds to a social movement that is over a decade old and influenced by many general trends in the digital economy and the information society [2]. Regardless of the form or purpose of eHealth resources, a common question is how HSP and HSU can optimally work together to design, build, and operationalize them; monitor their performance; and evaluate their impact [3].
Like most HSU, most HSP have little or no experience or training that equips them to collaborate effectively to develop eHealth resources, and so they are likely to turn to information technology professionals. However, technical developers or vendors who are commissioned to develop eHealth initiatives and technologies will turn back to their health sector partners for answers to who, what, where, when, why and how questions about engaging HSU in the early stages of the process. Furthermore, technical developers' responsibilities usually end on delivery of a working product. Thereafter, HSP may or may not have clear ways of assigning responsibility for managing and governing the product's use; in any case, HSU participation may be overlooked in these later stages in the life cycle of an eHealth resource. Participatory action research (PAR) [4] may be the launchpad for development [3], but at the end of development projects, there remains the need to operationalize and sustain the eHealth resources that have been created. The continuing quality assurance of eHealth resources within the auspicing health service also needs ongoing participation by HSU.
Apart from operational needs for HSU participation, there are ethical reasons for it. HSP have an ethical responsibility for ensuring that eHealth innovations achieve health outcomes for HSU. HSP are committed to evidence-based practice, in this as in other aspects of their work. Therefore, when they think about developing and deploying new eHealth resources, where do they find what is recognized as good practice in HSU participation? There are so many case studies that it is a near-impossible task to synthesize them all; furthermore, some talk the talk but do not walk the walk of HSU participation, some do not follow any recognized methodology, and some finish early in the life cycle of the eHealth resource.
There are numerous reviews and design guidelines that generalize about theories and methods of HSU participation in eHealth design. They emphasize the importance of the following basic principles: • appreciation and understanding from the outset, of the range of potential HSU characteristics, goals, needs, values, and perspectives on use [5][6][7] • attention to the needs of HSU not just as individual actors but also within their formal and informal care networks [8] • careful alignment of diverse concerns, attitudes, and perspectives that expert content creators, HSP, and HSU may have [9][10][11][12] • genuine active involvement so that HSU have an opportunity to identify practical problems and design, test, evaluate, and make decisions about technology in a range of environments [13][14][15] At the same time, they note that methods of HSU participation in eHealth design need to use human and other project resources judiciously. The themes are as follows: • ensuring that complex planning and evaluation models are able to be translated and streamlined to develop resources that are practical, feasible, and impactful in real-life settings [16] • taking a systematic approach to requirements specification to avoid mismatch with the organizational context and to support summative evaluation on a feature-specific level [17] • applying automation to expedite routine steps to create libraries of typical users and use cases and to manage unforeseen lessons learned for efficiency [18,19]

Definition Term
A state of complete physical, mental, and social well-being and not merely the absence of disease or infirmity, in accordance with the World Health Organization's definition [1] Health Organizations delivering support in the areas of physical, mental, or social well-being Health services Patients, clients, or consumers who are recipients of health services. Excludes family or informal caregivers who receive health services solely in their caring role Health service users Includes health professionals and health service managers Health service providers Health-related electronic resources, sometimes described as innovations, initiatives, applications, solutions, and tools Electronic health (eHealth) Interactive electronic resources including websites, portals, mobile technology, mobile apps, blogs, social media, podcasts, wearable fitness or tracking devices, e-learning, telehealth, video, electronic health records, and software eHealth resources Systematic methods used to include end users as codevelopers. Includes participatory approaches, processes, and frameworks, such as participatory action research Participatory methods Specific steps taken to ensure engagement of end users, such as focus groups Participatory processes Nevertheless, key considerations aside, it is difficult for HSP to identify from the literature a recognized, reliable methodological framework for engaging with HSU in the development of eHealth resources. A recent systematic review found that the literature variously encompassed 6 key phases and 17 different methods of participatory design, and it also found that sufficiency of reporting was poor and that no study undertook a robust assessment of efficacy [20]. This makes it difficult for HSP to study the effects of HSU participation in eHealth resources development on reach, adoption, acceptance, and efficacy of the intervention. Relative to other areas of health research, this type of study is immature, without widely endorsed methodological conventions for describing realistic aims for such projects or for determining valid measures of such effects [21]. Therefore, this paper investigates reports of eHealth applications and tools and resource development to determine what methods have been used systematically to ensure full HSU participation. We sought to distill evidence of positive, negative, or other unanticipated effects that have arisen at any stage in the eHealth resource life cycle from various HSU participation methods. Within these participatory approaches, we identified the reported impact from the point of view of HSU and HSP.
The impetus for this study began when the authors sought a strong research framework within which to undertake co-design of an eHealth initiative. The project was based on a print-based and workshop-based psychoeducational intervention called the Optimal Health Program (OHP). The authors wanted to ensure that they chose a rigorous methodological framework for redevelopment of OHP as an eHealth resource. Utilizing proven participatory methods would (1) optimize HSU engagement with the OHP resource that was developed, (2) strengthen the relevance of the resource to intended HSU, and (3) provide a logical foundation for long-term evaluation and improvement of the resource.

Objectives
This paper reviews published research reports that include detailed descriptions of participatory methods to engage HSU in eHealth resource development projects. Through synthesizing answers to the following questions, the objective of this paper is to support critical evaluation of this type of methodology and informed selection of appropriate approaches in future research and development projects:

Protocol and Registration
This systematic review has been carried out in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [22,23]. Protocol CRD42017053838 was lodged with the PROSPERO international prospective register of systematic reviews.

Information Sources
A total of 9 bibliographic databases were searched, including 6 health and biomedical databases and 3 technology databases: (codesign* or co-design* or coproduc* or co-produc* or cocreat* or co-creat* or participatory or e-collaboration or usability or focus group*).ti,ab.
3. ((user* or patient* or consumer* or family or families or carer* or caregiver* or participant* or client* or stakeholder* or peer*) adj2 (centre* or center* or centric or involv* or participat* or partner* or activat* or experience or advisor* or includ* or inclusion or engag* or collaborat* or consult* or empower* or input* or led or focus*)).ti,ab.
4. 1 or 2 or 3 5. internet/ or blogging/ or social media/ or audiovisual aids/ or multimedia/ or cell phones/ or text messaging/ or webcasts as topic/ or Telemedicine/ or videoconferencing/ or educational technology/ or audiovisual aids/ or motion pictures as topic/ or multimedia/ or exp optical storage devices/ or radio/ or exp tape recording/ or exp television/ or Mobile Applications/ or Software Design/ 6. (internet or web* or online or www* or audiovisual* or audio-visual* or multimedia or multi-media or ehealth or e-health or mobile tech* or mobile phone* or mobile device* or mobile health or mhealth or m-health or cell phone or cellular phone or smartphone or app or apps or blog* or social media or social network* or facebook or podcast* or tracking device* or electronic health device* or fitbit or elearning or e-learning or wearable device* or smartwatch* or wearable electronics or telemedicine or tele-medicine or telehealth or tele-health or video* or electronic patient record* or electronic medical record* or electronic health record* or electronic record* or wiki* or portal* or behavioural intervention technolog* or health information technolog* or software or medical informatic* or health informatic* or digital health).ti. 7. 5 or 6 8. Program development/ or planning techniques/ or equipment Design/ or software design/ 9. (develop* or creat* or plan* or build* or implement* or codesign* or co-design*).ti,ab. or design*.ti.

Study Selection
The search results were exported from all bibliographic databases to Endnote bibliographic management software.
Duplicates were identified and manually removed within Endnote by HW. The records were initially screened within Endnote on title and abstract by HW, excluding results that were clearly irrelevant, that is, not electronic technology, not health or well-being related, not development processes, or not involving end users. All potentially eligible records were exported from Endnote to Covidence, an online platform for managing the systematic review process. Covidence was used to screen records on title and abstract by any 2 of GM, HW, KG, and 1 additional reviewer using predefined inclusion and exclusion criteria as shown in Textboxes 2 and 3. All types of study design were eligible for initial inclusion.
Full-text articles were obtained and uploaded to Covidence for all the available records that had been included based on title and abstract. When a number of articles reported on the same project, they were grouped into a single study to be reviewed together. The full text was reviewed independently by 2 reviewers, GM and HW, using additional inclusion and exclusion criteria (Textboxes 4 and 5).

Data Collection Process
The included studies were found to use qualitative, quantitative, and mixed methods for HSU participation in eHealth resource development; therefore, the Mixed Methods Appraisal Tool (MMAT) Version 2011 [24] was selected to analyze the rigor of these studies. The MMAT's 19-assessment criteria were added to the extraction stage of Covidence. Each of the included full-text studies was assessed for methodological quality and rated according to the relevant MMAT criteria.
The detail of MMAT ratings under each criterion was extracted and recorded in an Excel spreadsheet. MMAT scoring metrics were used to calculate a total score for each study in order to develop a hierarchy of evidence for the strength of different methodologies (Multimedia Appendix 2).
The 2 reviewers, GM and HW, worked independently using MMAT to assess the methodological quality of papers and minimize risk of bias in assessing the literature. MMAT ratings and reasoning were compared, and conflicts were resolved through discussions between them.

Risk of Bias in Individual Studies
MMAT scores are typically 100%, 75%, 50%, and 25%. They work on the principle that a mixed-methods study is only as strong as its weakest part. This means that mixed-methods studies that have more criteria to meet (4 qualitative plus 4 quantitative plus 3 mixed method, equaling 11 criteria) could potentially be marked down more easily than studies that are purely qualitative and have fewer criteria to score (4 qualitative criteria only). In order to address this potential bias among study types, a decision was made to include an additional score of 90% to rationalize the difference that occurred between 100% and 75% in mixed-methods studies (Multimedia Appendix 2).
After assessment, studies were grouped by MMAT score and sorted into alphabetical order according to the surname of the first author. Although study numbers were initially used by the reviewers for identification purposes, these have been removed so that there is no confusion about study number and ranking. All studies with the same MMAT score hold equal ranking.

Data Items
The 90 studies assessed according to MMAT are summarized descriptively in a table (Multimedia Appendix 3). First, the data items described in Table 2 were manually extracted from the full text by HW and recorded in Excel for analysis. These results were grouped, tallied, and exported into separate tables according to characteristics of the research scope, such as health area, technology, population, or country of study (Multimedia Appendices 4-7).
Additional descriptive data were extracted from the full text of a subset of included studies, namely 30 studies that scored 90% or higher on MMAT. Data were extracted by HW and GM from the full text of each study using the data items listed in Table  3. These details were grouped, sorted, tallied, and exported into tables that summarize the main methods used to engage HSU in participatory development of eHealth resources.
Methods, frameworks, and processes varied enormously among studies; therefore, a decision was made to allocate all reported methods to the 5 stages of a single framework in order to standardize comparison. The Center for eHealth Research and Disease Management (CeHRes) Roadmap [25] was chosen for this purpose because it was specific to eHealth, highly cited (approximately 400 times between 2011 and 2017), based on the review of many eHealth and development frameworks, process oriented (not just a list of methods but a focus on specific steps), and defined within 5 stages.
Models and theories, participatory frameworks and interventions were extracted from the top 30 studies, and HW subsequently searched for additional mentions of them across the full text of the 90 MMAT-rated studies within Endnote.

Risk of Bias Across Studies
To minimize journal bias, a wide range of bibliographic databases were searched, including those with either a health focus or a technology focus. The search results were limited to English language, which could have created a cultural bias in the studies, although the 90 studies included in the quantitative analysis took place across 21 countries. Age group targeted by the resource (not to be confused by the age group of participants in the development process).
Simplified into 3 groups: youth (12-24 years), adult (25-64 years), and aged (65+ years). eHealth b projects aimed at children under 14 years were excluded; therefore, this age group was not included Age group Gender specific target of a resource; for example, female only or male only. Not recorded if the resource was inclusive of all genders rather than gender specific Gender specific LGBTQI+ specific target of a resource; for example, men who have sex with men. Not recorded if resource was inclusive rather than LGBTQI+ specific LGBTQI+ c specific Research focusing on a particular culture or across a number of different cultures (for example, Indigenous Australians). Not recorded if culture was not reported as an issue; for example, Swedish research taking place in Sweden with Swedishspeaking participants would not be included unless it was also researched in another country with another language for crosscultural comparison   The development of eHealth resources is a long process, sometimes taking many years, and many publications only reported a portion of the process, with only a few reporting the entire project up to final evaluation. As conference abstracts and grey literature were excluded in favor of journal articles, sections of the development process may have been reported elsewhere but not included in our evaluation. Reference lists and cited by references in Google Scholar were searched with respect to the top 90 studies to locate connected publications reporting later stages of development, but it is possible that some publications were either missed or published after our review timeframe.

Results
Database searches retrieved 24,674 records, which were exported to Endnote. Duplicates were removed by HW, leaving 15,117 records. These records were screened for broad relevance on title and abstract by HW and 13,096 records were excluded as clearly irrelevant. The remaining 2021 records were assessed for eligibility on title and abstract using the inclusion and exclusion criteria in Textboxes 2 and 3, and 1391 records were excluded.
The 630 remaining records were combined into 603 studies, some of which involved multiple publications. All 603 studies were assessed for eligibility on full text, and 513 studies were excluded according to the criteria in Textboxes 4 and 5, leaving 90 studies for quantitative analysis. During the screening and full text review process, 12 additional records relating to the 90 studies were identified from reference lists or contact with authors, and those records were combined into the studies. See Figure 1 for the PRISMA flow diagram.
A total of 90 studies were assessed for quality according to MMAT. Results are summarized in Table 4 and detailed results are available in Multimedia Appendix 2. An MMAT score of 100% was awarded to 28 studies and 2 studies scored 90%.

Results From 90 Studies Included in Quantitative Analysis
The 8 data items described in Table 2 were extracted from each of the 90 studies (Multimedia Appendix 3).
The major health focus of each study was grouped into a hierarchy of 18 wider MeSH subject headings, summarized in Multimedia Appendix 4. The top 5 health areas were neoplasms (cancer), mental disorders, nutritional and metabolic diseases (including weight management), virus diseases (including HIV), cardiovascular diseases, and endocrine system diseases (including diabetes).
Nine types of technology were reported in the 90 studies, and these are summarized in Multimedia Appendix 5. Websites (56 studies) and mobile apps (32 studies) were the main eHealth technologies developed. Other types of technology reported were decision tools, handheld computers, kiosk applications, personal health records, serious games, wearable devices, and telemonitoring.
Studies targeting specific populations are summarized in Multimedia Appendix 6. Of the 90 studies, 22 (24%) were youth specific, and 9 (10%) focused on the aged. Of the 90 studies, 11 (12%) reported eHealth projects for women only, and 4 (4%) were for men only. Moreover, 3 studies (3%) had a Lesbian, Gay, Bisexual, Transgender, Queer or Questioning, and Intersex+ focus. Fourteen studies (16%) had either a cultural or multicultural focus, such as a bilingual app for Indigenous Australians or the development of a website in both France and Finland.

Results From 30 Studies Included in Qualitative Synthesis
The 30 studies scoring 90% or higher on MMAT were recorded in Excel spreadsheets and reviewed in detail. Data items listed in Table 3 were extracted for each study (Multimedia Appendix 8).
The 30 studies are listed in Table 5, along with an indication of the CeHRes Roadmap stages reported. There was often a perceived overlap between stages 1 (contextual inquiry) and 2 (value specification) such as when focus groups may have covered both stages at once. Where this appeared to happen, it was reported in the spreadsheet and included in both stages in Table 5. Where the CeHRes Roadmap was particularly useful was in highlighting stages that were often not reported, such as operationalization or summative evaluation (Table 5). It is possible that some of these studies did address each stage but did not report them in journal articles that were reviewed.
A summary of the 30 highest MMAT-rated studies is represented in Table 6 with details of the product developed, technology used and targeted population. The health area and general purpose of each eHealth project, categorized using Medical Subject Headings (MeSH) is summarized in Multimedia Appendix 9.
The methods were recorded in Excel spreadsheets using the original terminology reported in each study. The details included the number of HSU or HSP involved in each process, the order of each activity as reported, and subprocesses within each method (for example, the type of design activity or workshop activity). These details are included in Multimedia Appendix 8. These detailed methods were then grouped so that they could be summarized using a consistent terminology and then compared. This summary of methods is included for each study in Table 7.
Models and theories referred to in the top 30 studies are shown in Table 8. Tables 9-13 give an overview of the options used to satisfy each stage of the CeHRes roadmap and the popularity of these methods. Many of the methods reported may demonstrate formative evaluation processes occurring as part of an iterative process. We recommend referring to Multimedia Appendix 8 and the original references for additional information that may be able to identify the practical steps that were implemented.  Lastly, the models and theories, participatory frameworks and interventions identified in the top 30 studies were searched across the full text of the 90 MMAT-rated studies within Endnote, and the results were ranked in order of prevalence in Multimedia Appendices 10-12.
Twenty three models or theories were identified from the 30 studies scoring 90% or higher on MMAT as playing a role in the development of eHealth resources. The most often reported models and theories were Social Cognitive Theory (n=4, 13%) [144], Theory of Planned Behavior (n=3, 10%) [145], Transtheoretical Model (Prochaska Stages of Change) (n=3, 10%) [146], and the Persuasive Technology Theory/Behavior Model for Persuasive Design (n=3, 10%) [147]. A large variety of other models and theories were referred to, with little overlap between studies. Full results are recorded in Multimedia Appendix 10.

Overview
In the era of digital health, we have a plethora of literature describing the need for better engagement with HSU to improve health care and health services, and we have access to the technologies to create a broad array of websites and mobile apps, but we lack detailed protocols for designing eHealth resources. This systematic review explored the participatory methods and frameworks used to engage HSU in the development of eHealth resources throughout the design process. UCD was most commonly reported but varied in its application and intention. Participatory methods promoting HSU engagement ranged from brief consultation via a review process to genuine collaboration, which included additional responsibility for the HSU in the actual creation process. Research and development projects that describe a conceptual model (such as Social Cognitive Theory) and a structured framework (such as the CeHRes Roadmap, which includes a diagram/flowchart) lay the foundations for us to gain greater insight into how particular processes lead to efficacious and effective eHealth resources.

Electronic Health Initiatives Developed and the Characteristics of Health Service Users
There have been extensive eHealth initiatives to address the issues of accessibility, engagement, health literacy, data collection, health promotion, early intervention, motivation, and behavioral change. Of the 90 MMAT-scored studies, websites and mobile apps make up the majority of eHealth initiatives presented in this review (Multimedia Appendix 5) with a strong multicultural focus (Multimedia Appendix 6). The end users of these eHealth initiatives were young adults, women, and the elderly (Multimedia Appendix 6) with the focus on cancer and mental health (Multimedia Appendix 4). The app has become an engagement tool used by HSP to make health information and health planning more interactive, interesting, and fun for HSU [30,32,35,36,42,43,56]. Moreover, participatory design is thought to enable young people to be creative and have substantial input into the resource development [30].

Participatory Frameworks
Analyzing the procedural frameworks used in our included studies, we found that no 2 studies reported their processes in the same way. The frameworks governing consumer participation were varied with the most reported being UCD, PAR Framework, CeHRes Roadmap, and MRC Guide to Developing and Evaluating Complex Interventions (Multimedia Appendix 11). The methods implemented to seek the HSU perspectives were also varied with the most reported being focus groups, surveys, interviews, prototype/storyboards, think aloud, and literature search (Tables 9-13). Theories and models that influenced procedures most commonly included cognition, behaviors, processes of change, motivation, and empowerment (Multimedia Appendix 10).
The diversity in eHealth initiatives supports creativity, and to ensure validity and strengthen eHealth research, there is a need to integrate a set of protocols for HSU participation and reporting guidelines [154] available via the Enhancing the QUAlity and Transparency Of health Research Network. This would not constrain methodological innovation and would allow a more effective meta-analysis and comparison of participatory development studies.

Methods Used in the Development of Resources
This review looked for evidence of sound methods for engaging HSU during the development of eHealth apps, tools, and resources. We found relatively few reports that described HSU participation throughout development (ie, from contextual enquiry to summative evaluation, Table 5). Furthermore, many of these reports did not provide adequate details according to mixed-methods appraisal standards. As shown in Multimedia Appendix 2, studies out of 603 full texts reviewed met all of our inclusion criteria and scored 90% or higher according to MMAT. This suggests that research training, funding, and dissemination agencies need to attach far greater importance to reports that describe methods more rigorously.
Others have observed that "The diverse communities working in digital health-including government stakeholders, technologists, clinicians, implementers, network operators, researchers, donors-have lacked a mutually understandable language with which to assess and articulate functionality" [155]. Tables 9-13 illustrates how deeply this lack has affected the production of cohesive research evidence, that is, it is virtually impossible to map the semantic relationships among the methodology elements to inform the discourse about what forms of participatory eHealth design work and why. Many methods are generic to human computer interaction, some take a broad behavioral approach and some include methods of measuring health outcomes in the particular area of health where the intervention is directed. One possible view is that this illustrates a flourishing of innovation and creativity. Another is that this creates a minefield for research training and peer reviewing and may represent a considerable waste of research resources.
Analyzing the conceptual bases for the methods used in the 30 studies scoring 90% or higher on MMAT, we found much variety with 23 different models or theories reported (Multimedia Appendix 10). The most commonly occurring theories were Social Cognitive Theory, Theory of Planned Behavior, Transtheoretical Model, Persuasive Technology Theory, and Health Behavior Theory. This finding offers a sound basis in evidence for future researchers who wish to follow these precedents. However, we note that research in this area has not been informed by other potentially relevant theories (for example, theories that may account better for healthcare consumers' economic, emotional, or empowerment motives for engagement) [156].

Effective Involvement of Health Service Users
This review looked for evidence about the effectiveness of particular approaches in terms of supporting involvement by HSU. Winterling reported strategies implemented to address engagement with HSU, including 1-person central contact, established expectation of roles, compensation for time, reaching a common agreement, and HSU seen as experts on patient perspective [63][64][65][66].
It is also possible to reflect on the richness of the findings generated by particular approaches. As shown in Multimedia Appendix 13, each study reported between 2 and 10 major thematic outputs. Reports with relatively concise outputs were Bengtsson [28,29] using participatory research design and O'Brien [51] describing an array of approaches. The most extensive review was reported by Fleisher [68] using the Ottawa Decision Support Framework and participatory design and Goldenberg [42,43] using 3 types of iterative qualitative research approaches. In assessing effectiveness this way, unknown factors may be in play, such as sophistication of the data collection procedures, analytical expertise of the researchers, editorial constraints on reporting results, and temporal pressures on publication.

Important Aspects of Participatory Methods for Health Service Users
There were consistent themes that represented HSU priorities in eHealth initiatives across the selected 30 studies represented in Multimedia Appendix 13. Access to relevant, simple, and clear health information was reported consistently across most of the studies highlighting the importance of this information to make informed decisions in a timely manner. A well-designed eHealth resource that includes a framework supporting HSU involvement can significantly impact health literacy for both HSU and HSP. HSU involvement with the development of an eHealth resource created a collaborative process that required transparency and respect as well as clear mediation processes [53][54][55].
Being involved in the development of an eHealth resource created the opportunity for HSU to clarify the user perspective and support the relevance of the final product. Despite the variety of websites and apps, HSU reported the need for improved access to information, coordination of care, interactivity with information provided, culturally specific information, patient education, and self-management. HSU also acknowledged the importance of confidentiality and privacy when exchanging personal health information over electronic networks.

Impact of Participatory Methods Reported by Researchers
The researchers reported a number of key issues highlighting the importance of participatory methods in creating an eHealth resource that was relevant to HSU. In Multimedia Appendix 13, an outline of the research recommendations was documented for the selected 30 studies. Researchers reported on the importance of utilizing a participatory design, which included an iterative process that increased the responsiveness and relevance of their eHealth initiatives. Having the HSU perspective from the beginning was important as well as ensuring that the process was genuinely collaborative with all participants respected and acknowledged. Utilizing a health behavior theory in combination with a participatory design was noted to enhance the eHealth resource. The theory base acknowledges the importance of motivation, empowerment, and stages of change in supporting the engagement and utilization of the eHealth resource. It was also noted that the eHealth resource needed to be interesting, engaging, and in some instances include a game-playing element. Creating a more positive approach enabled the HSU to engage with serious and difficult health issues and explore options for improved health. Not only did the eHealth resource need to be interesting but it also importantly needed to be intuitive and simple to navigate.
The demand on time and financial resources to implement a participatory design was noted by some researchers [27,30,42,43,57,68]. Availability of resources was an important consideration throughout the design process, which was often iterative. With the rapid change in technology, there is an increasing demand for HSP to be agile and develop eHealth resources more quickly but still maintaining an evidence-based, best-practice approach inclusive of HSU participation.

Limitations
A limitation of our final dataset is that because of the number of papers retrieved, we decided to limit our analysis to published journal articles and to leave out full papers in conference proceedings. It is possible that there are strong participatory processes that have not been reported in detail, or at all, in the journal literature. Moreover, we did not include studies published in languages other than English and therefore we cannot be certain that our dataset reflects work being done around the world.
As our focus was on the inclusion of HSU from the early development process onward, some studies were included that did not extend to a final evaluation of the product, and it was not always possible to consider the success or otherwise of the final eHealth product. As a part of our inclusion criteria, we required some evidence that a specific eHealth product was ultimately created or likely to be taken to completion.
A limitation of our data analysis is that MMAT is a critical appraisal tool to assess the methodological quality of studies. It does not assess the quality of the writing or the content of the research; therefore, it is possible that we have overlooked papers that may be of high quality in other respects but which we have not rated highly here because of the way their methods sections are presented. For example, under MMAT, a paper will not score highly if it does not discuss the impact of the research or report the limitations of a mixed-methods study. The studies may not have rated highly under MMAT if they used both qualitative and quantitative methods but did not acknowledge that this constituted a mixed-methods study or if only selected aspects were reported. For example, a study that reported HSU participation only at the summative evaluation stage may have involved HSU earlier as well, but this would not register in our search results because we looked for descriptions of methods for HSU participation from initial design stages.
Although categorizing all reported methods in these studies according to the 5 stages of the CeHRes Roadmap [25] was a generally useful way to compare processes across studies, absolute consistency was not achievable because of the wide variety of structuring reports, the differing terminology and naming conventions used for similar methods, and the difficulty in allocating all methods accurately to a particular process stage.

Conclusions
Agility of eHealth development is problematic in comparison to nonmedical industries as we seek to ensure safety and quality of care for HSU. It is a challenge for eHealth development to follow rigorous methods within a timeframe that responds to current needs, limited resources, and rapid technological changes. Methodological approaches to developing eHealth resources vary but the importance of engaging HSU in participatory design is consistently emphasized. By synthesizing the existing evidence about strong mixed methods for participatory development of eHealth resources, we anticipate that this systematic review will provide others with clearer guidance to plan more rapid and better-structured work of this kind.

Conflicts of Interest
None declared.

Multimedia Appendix 1
Search strategies for all databases.

Multimedia Appendix 7
Countries where study took place in 90 studies.

Multimedia Appendix 8
Detail of data extracted from 30 studies.

Multimedia Appendix 9
Health area and purpose of product in top 30 studies categorised using Medical Subject Headings (MeSH).

Multimedia Appendix 10
Model or theory base in top 30 studies.

Multimedia Appendix 11
Participatory frameworks and approaches in top 30 studies.

Multimedia Appendix 12
Interventions in top 30 studies.