Individuals are increasingly being offered access to health services via electronic health (eHealth), sometimes called digital health, that is, health-related electronic resources that connect them with health service providers (HSP) over the internet. Examples include websites, portals, social media sites, serious games, mobile apps, wearable self-monitoring devices, online learning sites, telehealth platforms, and shareable electronic health records. Patients, clients, or consumers in this review are called health service users (HSU). They may require services to support their physical health, mental health, and well-being in the broadest sense of the World Health Organization’s definition (Table 1) [1].

The involvement of HSU as full participants in eHealth innovations responds to a social movement that is over a decade old and influenced by many general trends in the digital economy and the information society [2]. Regardless of the form or purpose of eHealth resources, a common question is how HSP and HSU can optimally work together to design, build, and operationalize them; monitor their performance; and evaluate their impact [3].

Like most HSU, most HSP have little or no experience or training that equips them to collaborate effectively to develop eHealth resources, and so they are likely to turn to information technology professionals. However, technical developers or vendors who are commissioned to develop eHealth initiatives and technologies will turn back to their health sector partners for answers to who, what, where, when, why and how questions about engaging HSU in the early stages of the process. Furthermore, technical developers’ responsibilities usually end on delivery of a working product. Thereafter, HSP may or may not have clear ways of assigning responsibility for managing and governing the product’s use; in any case, HSU participation may be overlooked in these later stages in the life cycle of an eHealth resource. Participatory action research (PAR) [4] may be the launchpad for development [3], but at the end of development projects, there remains the need to operationalize and sustain the eHealth resources that have been created. The continuing quality assurance of eHealth resources within the auspicing health service also needs ongoing participation by HSU.

Apart from operational needs for HSU participation, there are ethical reasons for it. HSP have an ethical responsibility for ensuring that eHealth innovations achieve health outcomes for HSU. HSP are committed to evidence-based practice, in this as in other aspects of their work. Therefore, when they think about developing and deploying new eHealth resources, where do they find what is recognized as good practice in HSU participation? There are so many case studies that it is a near-impossible task to synthesize them all; furthermore, some talk the talk but do not walk the walk of HSU participation, some do not follow any recognized methodology, and some finish early in the life cycle of the eHealth resource.

There are numerous reviews and design guidelines that generalize about theories and methods of HSU participation in eHealth design. They emphasize the importance of the following basic principles:

At the same time, they note that methods of HSU participation in eHealth design need to use human and other project resources judiciously. The themes are as follows:

Nevertheless, key considerations aside, it is difficult for HSP to identify from the literature a recognized, reliable methodological framework for engaging with HSU in the development of eHealth resources. A recent systematic review found that the literature variously encompassed 6 key phases and 17 different methods of participatory design, and it also found that sufficiency of reporting was poor and that no study undertook a robust assessment of efficacy [20]. This makes it difficult for HSP to study the effects of HSU participation in eHealth resources development on reach, adoption, acceptance, and efficacy of the intervention. Relative to other areas of health research, this type of study is immature, without widely endorsed methodological conventions for describing realistic aims for such projects or for determining valid measures of such effects [21].

Therefore, this paper investigates reports of eHealth applications and tools and resource development to determine what methods have been used systematically to ensure full HSU participation. We sought to distill evidence of positive, negative, or other unanticipated effects that have arisen at any stage in the eHealth resource life cycle from various HSU participation methods. Within these participatory approaches, we identified the reported impact from the point of view of HSU and HSP.

The impetus for this study began when the authors sought a strong research framework within which to undertake co-design of an eHealth initiative. The project was based on a print-based and workshop-based psychoeducational intervention called the Optimal Health Program (OHP). The authors wanted to ensure that they chose a rigorous methodological framework for redevelopment of OHP as an eHealth resource. Utilizing proven participatory methods would (1) optimize HSU engagement with the OHP resource that was developed, (2) strengthen the relevance of the resource to intended HSU, and (3) provide a logical foundation for long-term evaluation and improvement of the resource.

This paper reviews published research reports that include detailed descriptions of participatory methods to engage HSU in eHealth resource development projects. Through synthesizing answers to the following questions, the objective of this paper is to support critical evaluation of this type of methodology and informed selection of appropriate approaches in future research and development projects:

This systematic review has been carried out in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [22,23]. Protocol CRD42017053838 was lodged with the PROSPERO international prospective register of systematic reviews.

A total of 9 bibliographic databases were searched, including 6 health and biomedical databases and 3 technology databases:

Additional articles were identified from reference lists of key articles and cited by references in Google Scholar.

Search strategies were developed by an experienced medical research librarian (HW) in consultation with the OHP project leader (GM) and an expert eHealth researcher (KG).

In December 2015, scoping searches were developed and run in MEDLINE, EMBASE, CINAHL, PsycINFO, and Cochrane. In April 2016, brief confirmatory searches were run in Google Scholar to consider gaps in the initial strategy and additional search terms or databases that could be included. As a result, search strategies were refined and rerun in the initial health and biomedical databases as well as 3 additional technology databases: Web of Science, ACM, and IEEE. In February 2017, searches were updated to include results to the end of 2016. At this stage, an additional health database, Emcare, was also searched.

Within the health and biomedical databases (MEDLINE, EMBASE, CINAHL, PsycINFO, Emcare, and Cochrane) the search strategies combined the general concepts of user participation AND electronic resources AND program design. These search strategies were not limited to health-related conditions or resources because they yielded a small proportion of nonhealth-related results that could be removed manually. This enabled a very wide range of health conditions, HSU, organizations, and resources to be included in the results.

Within the broader technology databases that are not health specific (Web of Science, ACM, and IEEE), the search strategies were necessarily limited to health-related resources, combining the general concepts of user participation AND electronic resources AND (health OR well-being) AND program design.

A detailed search strategy was developed for MEDLINE using a combination of Medical Subject Headings (MeSH) and text words (Textbox 1). This was then adapted for the other databases, taking into account relevant subject headings and syntax. Search results were limited to publications dated from January 2006 to December 2016 and publications in English language. All database searches were updated in February 2017. Final search strategies for all databases are provided in Multimedia Appendix 1.

The search results were exported from all bibliographic databases to Endnote bibliographic management software. Duplicates were identified and manually removed within Endnote by HW. The records were initially screened within Endnote on title and abstract by HW, excluding results that were clearly irrelevant, that is, not electronic technology, not health or well-being related, not development processes, or not involving end users. All potentially eligible records were exported from Endnote to Covidence, an online platform for managing the systematic review process. Covidence was used to screen records on title and abstract by any 2 of GM, HW, KG, and 1 additional reviewer using predefined inclusion and exclusion criteria as shown in Textboxes 2 and 3. All types of study design were eligible for initial inclusion.

Full-text articles were obtained and uploaded to Covidence for all the available records that had been included based on title and abstract. When a number of articles reported on the same project, they were grouped into a single study to be reviewed together. The full text was reviewed independently by 2 reviewers, GM and HW, using additional inclusion and exclusion criteria (Textboxes 4 and 5).

The included studies were found to use qualitative, quantitative, and mixed methods for HSU participation in eHealth resource development; therefore, the Mixed Methods Appraisal Tool (MMAT) Version 2011 [24] was selected to analyze the rigor of these studies. The MMAT’s 19-assessment criteria were added to the extraction stage of Covidence. Each of the included full-text studies was assessed for methodological quality and rated according to the relevant MMAT criteria.

The detail of MMAT ratings under each criterion was extracted and recorded in an Excel spreadsheet. MMAT scoring metrics were used to calculate a total score for each study in order to develop a hierarchy of evidence for the strength of different methodologies (Multimedia Appendix 2).

The 2 reviewers, GM and HW, worked independently using MMAT to assess the methodological quality of papers and minimize risk of bias in assessing the literature. MMAT ratings and reasoning were compared, and conflicts were resolved through discussions between them.

The 8 data items described in Table 2 were extracted from each of the 90 studies (Multimedia Appendix 3).

The major health focus of each study was grouped into a hierarchy of 18 wider MeSH subject headings, summarized in Multimedia Appendix 4. The top 5 health areas were neoplasms (cancer), mental disorders, nutritional and metabolic diseases (including weight management), virus diseases (including HIV), cardiovascular diseases, and endocrine system diseases (including diabetes).

Nine types of technology were reported in the 90 studies, and these are summarized in Multimedia Appendix 5. Websites (56 studies) and mobile apps (32 studies) were the main eHealth technologies developed. Other types of technology reported were decision tools, handheld computers, kiosk applications, personal health records, serious games, wearable devices, and telemonitoring.

Studies targeting specific populations are summarized in Multimedia Appendix 6. Of the 90 studies, 22 (24%) were youth specific, and 9 (10%) focused on the aged. Of the 90 studies, 11 (12%) reported eHealth projects for women only, and 4 (4%) were for men only. Moreover, 3 studies (3%) had a Lesbian, Gay, Bisexual, Transgender, Queer or Questioning, and Intersex+ focus. Fourteen studies (16%) had either a cultural or multicultural focus, such as a bilingual app for Indigenous Australians or the development of a website in both France and Finland.

The 90 studies took place in 21 countries, summarized in Multimedia Appendix 7. The top 6 countries were United States (33 studies), United Kingdom (15 studies), Netherlands (13 studies), Canada (7 studies), Sweden (6 studies), and Australia (6 studies). Studies also took place in Austria, Belgium, Czech Republic, Greece, Denmark, Finland, France, India, Spain, Ireland, Italy, New Zealand, Norway, Republic of Korea, and Saudi Arabia.

The 30 studies scoring 90% or higher on MMAT were recorded in Excel spreadsheets and reviewed in detail. Data items listed in Table 3 were extracted for each study (Multimedia Appendix 8).

The 30 studies are listed in Table 5, along with an indication of the CeHRes Roadmap stages reported. There was often a perceived overlap between stages 1 (contextual inquiry) and 2 (value specification) such as when focus groups may have covered both stages at once. Where this appeared to happen, it was reported in the spreadsheet and included in both stages in Table 5. Where the CeHRes Roadmap was particularly useful was in highlighting stages that were often not reported, such as operationalization or summative evaluation (Table 5). It is possible that some of these studies did address each stage but did not report them in journal articles that were reviewed.

A summary of the 30 highest MMAT–rated studies is represented in Table 6 with details of the product developed, technology used and targeted population. The health area and general purpose of each eHealth project, categorized using Medical Subject Headings (MeSH) is summarized in Multimedia Appendix 9.

The methods were recorded in Excel spreadsheets using the original terminology reported in each study. The details included the number of HSU or HSP involved in each process, the order of each activity as reported, and subprocesses within each method (for example, the type of design activity or workshop activity). These details are included in Multimedia Appendix 8. These detailed methods were then grouped so that they could be summarized using a consistent terminology and then compared. This summary of methods is included for each study in Table 7.

Models and theories referred to in the top 30 studies are shown in Table 8.

Tables 9-13 give an overview of the options used to satisfy each stage of the CeHRes roadmap and the popularity of these methods. Many of the methods reported may demonstrate formative evaluation processes occurring as part of an iterative process. We recommend referring to Multimedia Appendix 8 and the original references for additional information that may be able to identify the practical steps that were implemented.

Lastly, the models and theories, participatory frameworks and interventions identified in the top 30 studies were searched across the full text of the 90 MMAT–rated studies within Endnote, and the results were ranked in order of prevalence in Multimedia Appendices 10-12.

Twenty three models or theories were identified from the 30 studies scoring 90% or higher on MMAT as playing a role in the development of eHealth resources. The most often reported models and theories were Social Cognitive Theory (n=4, 13%) [144], Theory of Planned Behavior (n=3, 10%) [145], Transtheoretical Model (Prochaska Stages of Change) (n=3, 10%) [146], and the Persuasive Technology Theory/Behavior Model for Persuasive Design (n=3, 10%) [147]. A large variety of other models and theories were referred to, with little overlap between studies. Full results are recorded in Multimedia Appendix 10.

A total of 24 named participatory frameworks or approaches were identified from the 30 studies scoring 90% or higher on MMAT (Multimedia Appendix 11). Only 20 of the 30 top scoring studies referred to a specific framework, with many studies referring more broadly to using participatory design or iterative design methods without reference to a particular named framework. The most often reported participatory frameworks or approaches were User-Centered Design ([UCD], n=5, 17%) [148], PAR framework (n=4, 13%) [149], CeHRes Roadmap (n=3, 10%) [25], Medical Research Council (MRC) Guide to Developing and Evaluating Complex Interventions (n=2, 7%) [150-152], and International Patient Decision Aid Standards Collaboration (n=2, 7%) [153].

Some studies referred to specific interventions that were integral to the function of the eHealth resource that was developed. Key interventions identified in Multimedia Appendix 12 were Cognitive Behavior Therapy (CBT), Behavior Change Techniques, and Mindfulness.

In the era of digital health, we have a plethora of literature describing the need for better engagement with HSU to improve health care and health services, and we have access to the technologies to create a broad array of websites and mobile apps, but we lack detailed protocols for designing eHealth resources. This systematic review explored the participatory methods and frameworks used to engage HSU in the development of eHealth resources throughout the design process. UCD was most commonly reported but varied in its application and intention. Participatory methods promoting HSU engagement ranged from brief consultation via a review process to genuine collaboration, which included additional responsibility for the HSU in the actual creation process. Research and development projects that describe a conceptual model (such as Social Cognitive Theory) and a structured framework (such as the CeHRes Roadmap, which includes a diagram/flowchart) lay the foundations for us to gain greater insight into how particular processes lead to efficacious and effective eHealth resources.

There have been extensive eHealth initiatives to address the issues of accessibility, engagement, health literacy, data collection, health promotion, early intervention, motivation, and behavioral change. Of the 90 MMAT-scored studies, websites and mobile apps make up the majority of eHealth initiatives presented in this review (Multimedia Appendix 5) with a strong multicultural focus (Multimedia Appendix 6). The end users of these eHealth initiatives were young adults, women, and the elderly (Multimedia Appendix 6) with the focus on cancer and mental health (Multimedia Appendix 4). The app has become an engagement tool used by HSP to make health information and health planning more interactive, interesting, and fun for HSU [30,32,35,36,42,43,56]. Moreover, participatory design is thought to enable young people to be creative and have substantial input into the resource development [30].

Analyzing the procedural frameworks used in our included studies, we found that no 2 studies reported their processes in the same way. The frameworks governing consumer participation were varied with the most reported being UCD, PAR Framework, CeHRes Roadmap, and MRC Guide to Developing and Evaluating Complex Interventions (Multimedia Appendix 11). The methods implemented to seek the HSU perspectives were also varied with the most reported being focus groups, surveys, interviews, prototype/storyboards, think aloud, and literature search (Tables 9-13). Theories and models that influenced procedures most commonly included cognition, behaviors, processes of change, motivation, and empowerment (Multimedia Appendix 10).

The diversity in eHealth initiatives supports creativity, and to ensure validity and strengthen eHealth research, there is a need to integrate a set of protocols for HSU participation and reporting guidelines [154] available via the Enhancing the QUAlity and Transparency Of health Research Network. This would not constrain methodological innovation and would allow a more effective meta-analysis and comparison of participatory development studies.

This review looked for evidence of sound methods for engaging HSU during the development of eHealth apps, tools, and resources. We found relatively few reports that described HSU participation throughout development (ie, from contextual enquiry to summative evaluation, Table 5). Furthermore, many of these reports did not provide adequate details according to mixed-methods appraisal standards. As shown in Multimedia Appendix 2, studies out of 603 full texts reviewed met all of our inclusion criteria and scored 90% or higher according to MMAT. This suggests that research training, funding, and dissemination agencies need to attach far greater importance to reports that describe methods more rigorously.

Others have observed that “The diverse communities working in digital health—including government stakeholders, technologists, clinicians, implementers, network operators, researchers, donors—have lacked a mutually understandable language with which to assess and articulate functionality” [155]. Tables 9-13 illustrates how deeply this lack has affected the production of cohesive research evidence, that is, it is virtually impossible to map the semantic relationships among the methodology elements to inform the discourse about what forms of participatory eHealth design work and why. Many methods are generic to human computer interaction, some take a broad behavioral approach and some include methods of measuring health outcomes in the particular area of health where the intervention is directed. One possible view is that this illustrates a flourishing of innovation and creativity. Another is that this creates a minefield for research training and peer reviewing and may represent a considerable waste of research resources.

Analyzing the conceptual bases for the methods used in the 30 studies scoring 90% or higher on MMAT, we found much variety with 23 different models or theories reported (Multimedia Appendix 10). The most commonly occurring theories were Social Cognitive Theory, Theory of Planned Behavior, Transtheoretical Model, Persuasive Technology Theory, and Health Behavior Theory. This finding offers a sound basis in evidence for future researchers who wish to follow these precedents. However, we note that research in this area has not been informed by other potentially relevant theories (for example, theories that may account better for healthcare consumers’ economic, emotional, or empowerment motives for engagement) [156].

This review looked for evidence about the effectiveness of particular approaches in terms of supporting involvement by HSU. Winterling reported strategies implemented to address engagement with HSU, including 1-person central contact, established expectation of roles, compensation for time, reaching a common agreement, and HSU seen as experts on patient perspective [63-66].

It is also possible to reflect on the richness of the findings generated by particular approaches. As shown in Multimedia Appendix 13, each study reported between 2 and 10 major thematic outputs. Reports with relatively concise outputs were Bengtsson [28,29] using participatory research design and O’Brien [51] describing an array of approaches. The most extensive review was reported by Fleisher [68] using the Ottawa Decision Support Framework and participatory design and Goldenberg [42,43] using 3 types of iterative qualitative research approaches. In assessing effectiveness this way, unknown factors may be in play, such as sophistication of the data collection procedures, analytical expertise of the researchers, editorial constraints on reporting results, and temporal pressures on publication.

There were consistent themes that represented HSU priorities in eHealth initiatives across the selected 30 studies represented in Multimedia Appendix 13. Access to relevant, simple, and clear health information was reported consistently across most of the studies highlighting the importance of this information to make informed decisions in a timely manner. A well-designed eHealth resource that includes a framework supporting HSU involvement can significantly impact health literacy for both HSU and HSP. HSU involvement with the development of an eHealth resource created a collaborative process that required transparency and respect as well as clear mediation processes [53-55].

Being involved in the development of an eHealth resource created the opportunity for HSU to clarify the user perspective and support the relevance of the final product. Despite the variety of websites and apps, HSU reported the need for improved access to information, coordination of care, interactivity with information provided, culturally specific information, patient education, and self-management. HSU also acknowledged the importance of confidentiality and privacy when exchanging personal health information over electronic networks.

The researchers reported a number of key issues highlighting the importance of participatory methods in creating an eHealth resource that was relevant to HSU. In Multimedia Appendix 13, an outline of the research recommendations was documented for the selected 30 studies. Researchers reported on the importance of utilizing a participatory design, which included an iterative process that increased the responsiveness and relevance of their eHealth initiatives. Having the HSU perspective from the beginning was important as well as ensuring that the process was genuinely collaborative with all participants respected and acknowledged. Utilizing a health behavior theory in combination with a participatory design was noted to enhance the eHealth resource. The theory base acknowledges the importance of motivation, empowerment, and stages of change in supporting the engagement and utilization of the eHealth resource. It was also noted that the eHealth resource needed to be interesting, engaging, and in some instances include a game-playing element. Creating a more positive approach enabled the HSU to engage with serious and difficult health issues and explore options for improved health. Not only did the eHealth resource need to be interesting but it also importantly needed to be intuitive and simple to navigate.

Heckman [45] reported that their eHealth initiative was guided by intervention development, assessment guidelines for behavioral therapy, and health communication programs with health literacy best-practice. Utilizing a participatory design appeared to improve the relevance of the eHealth resource by addressing issues of culture, gender, age, and sexuality (Multimedia Appendix 6). Goldenberg [42,43] reported personalization along with interactive functionality promoted ownership for HSU. A majority of projects included both HSU and HSP in participatory methods across different developmental stages from contextual inquiry to summative evaluation of the project [28,29,31,34-38,40-43,45-55,58-61,63-67,69]. Evaluation is an integral part of participatory methodology; however, this was reported inconsistently across the 30 studies (Multimedia Appendix 8). The inclusion of a standardized tool to evaluate processes and outcomes from the HSU perspective, as part of a participatory framework, may address the need to bring more objectivity to evaluating various studies.

The demand on time and financial resources to implement a participatory design was noted by some researchers [27,30,42,43,57,68]. Availability of resources was an important consideration throughout the design process, which was often iterative. With the rapid change in technology, there is an increasing demand for HSP to be agile and develop eHealth resources more quickly but still maintaining an evidence-based, best-practice approach inclusive of HSU participation.

A limitation of our final dataset is that because of the number of papers retrieved, we decided to limit our analysis to published journal articles and to leave out full papers in conference proceedings. It is possible that there are strong participatory processes that have not been reported in detail, or at all, in the journal literature. Moreover, we did not include studies published in languages other than English and therefore we cannot be certain that our dataset reflects work being done around the world.

As our focus was on the inclusion of HSU from the early development process onward, some studies were included that did not extend to a final evaluation of the product, and it was not always possible to consider the success or otherwise of the final eHealth product. As a part of our inclusion criteria, we required some evidence that a specific eHealth product was ultimately created or likely to be taken to completion.

A limitation of our data analysis is that MMAT is a critical appraisal tool to assess the methodological quality of studies. It does not assess the quality of the writing or the content of the research; therefore, it is possible that we have overlooked papers that may be of high quality in other respects but which we have not rated highly here because of the way their methods sections are presented. For example, under MMAT, a paper will not score highly if it does not discuss the impact of the research or report the limitations of a mixed-methods study. The studies may not have rated highly under MMAT if they used both qualitative and quantitative methods but did not acknowledge that this constituted a mixed-methods study or if only selected aspects were reported. For example, a study that reported HSU participation only at the summative evaluation stage may have involved HSU earlier as well, but this would not register in our search results because we looked for descriptions of methods for HSU participation from initial design stages.

Although categorizing all reported methods in these studies according to the 5 stages of the CeHRes Roadmap [25] was a generally useful way to compare processes across studies, absolute consistency was not achievable because of the wide variety of structuring reports, the differing terminology and naming conventions used for similar methods, and the difficulty in allocating all methods accurately to a particular process stage.

Agility of eHealth development is problematic in comparison to nonmedical industries as we seek to ensure safety and quality of care for HSU. It is a challenge for eHealth development to follow rigorous methods within a timeframe that responds to current needs, limited resources, and rapid technological changes. Methodological approaches to developing eHealth resources vary but the importance of engaging HSU in participatory design is consistently emphasized. By synthesizing the existing evidence about strong mixed methods for participatory development of eHealth resources, we anticipate that this systematic review will provide others with clearer guidance to plan more rapid and better-structured work of this kind.