Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, March 11, 2019 at 4:00 PM to 4:30 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 22.02.19 in Vol 11, No 1 (2019): Jan-Mar

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/11474, first published Jul 04, 2018.

This paper is in the following e-collection/theme issue:

    Review

    Participatory Methods to Engage Health Service Users in the Development of Electronic Health Resources: Systematic Review

    1Mental Health Executive Services, St Vincent's Hospital, Melbourne, Fitzroy, Australia

    2Department of Nursing, Faculty of Medicine, Dentistry and Health Sciences, University of Melbourne, Melbourne, Australia

    3Library Service, St Vincent's Hospital Melbourne, Fitzroy, Australia

    4Health and Biomedical Informatics Centre, University of Melbourne, Melbourne, Australia

    5Department of Psychiatry, Faculty of Medicine, Dentistry and Health Sciences, University of Melbourne, Melbourne, Australia

    *these authors contributed equally

    Corresponding Author:

    Gaye Moore, BN (Hons), MPH, PhD

    Mental Health Executive Services

    St Vincent's Hospital, Melbourne

    PO Box 2900

    Fitzroy, 3065

    Australia

    Phone: 61 392311938

    Email: gaye.moore@svha.org.au


    ABSTRACT

    Background: When health service providers (HSP) plan to develop electronic health (eHealth) resources for health service users (HSU), the latter’s involvement is essential. Typically, however, HSP, HSU, and technology developers engaged to produce the resources lack expertise in participatory design methodologies suited to the eHealth context. Furthermore, it can be difficult to identify an established method to use, or determine how to work stepwise through any particular process.

    Objective: We sought to summarize the evidence about participatory methods and frameworks used to engage HSU in the development of eHealth resources from the beginning of the design process.

    Methods: We searched for studies reporting participatory processes in initial development of eHealth resources from 2006 to 2016 in 9 bibliographic databases: MEDLINE, EMBASE, CINAHL, PsycINFO, Emcare, Cochrane Library, Web of Science, ACM Guide to Computing Literature, and IEEE Xplore. From 15,117 records initially screened on title and abstract for relevance to eHealth and early participatory design, 603 studies were assessed for eligibility on full text. The remaining 90 studies were rated by 2 reviewers using the Mixed Methods Appraisal Tool Version 2011 (Pluye et al; MMAT) and analyzed with respect to health area, purpose, technology type, and country of study. The 30 studies scoring 90% or higher on MMAT were included in a detailed qualitative synthesis.

    Results: Of the 90 MMAT-rated studies, the highest reported (1) health areas were cancer and mental disorders, (2) eHealth technologies were websites and mobile apps, (3) targeted populations were youth and women, and (4) countries of study were the United States, the United Kingdom, and the Netherlands. Of the top 30 studies the highest reported participatory frameworks were User-Centered Design, Participatory Action Research Framework, and the Center for eHealth Research and Disease Management (CeHRes) Roadmap, and the highest reported model underpinning development and engagement was Social Cognitive Theory. Of the 30 studies, 4 reported on all the 5 stages of the CeHRes Roadmap.

    Conclusions: The top 30 studies yielded 24 participatory frameworks. Many studies referred to using participatory design methods without reference to a framework. The application of a structured framework such as the CeHRes Roadmap and a model such as Social Cognitive Theory creates a foundation for a well-designed eHealth initiative that ensures clarity and enables replication across participatory design projects. The framework and model need to be clearly articulated and address issues that include resource availability, responsiveness to change, and the criteria for good practice. This review creates an information resource for future eHealth developers, to guide the design of their eHealth resource with a framework that can support further evaluation and development.

    Trial Registration: PROSPERO CRD42017053838; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=53838

    J Participat Med 2019;11(1):e11474

    doi:10.2196/11474

    KEYWORDS



    Introduction

    Rationale

    Individuals are increasingly being offered access to health services via electronic health (eHealth), sometimes called digital health, that is, health-related electronic resources that connect them with health service providers (HSP) over the internet. Examples include websites, portals, social media sites, serious games, mobile apps, wearable self-monitoring devices, online learning sites, telehealth platforms, and shareable electronic health records. Patients, clients, or consumers in this review are called health service users (HSU). They may require services to support their physical health, mental health, and well-being in the broadest sense of the World Health Organization’s definition (Table 1) [1].

    The involvement of HSU as full participants in eHealth innovations responds to a social movement that is over a decade old and influenced by many general trends in the digital economy and the information society [2]. Regardless of the form or purpose of eHealth resources, a common question is how HSP and HSU can optimally work together to design, build, and operationalize them; monitor their performance; and evaluate their impact [3].

    Like most HSU, most HSP have little or no experience or training that equips them to collaborate effectively to develop eHealth resources, and so they are likely to turn to information technology professionals. However, technical developers or vendors who are commissioned to develop eHealth initiatives and technologies will turn back to their health sector partners for answers to who, what, where, when, why and how questions about engaging HSU in the early stages of the process. Furthermore, technical developers’ responsibilities usually end on delivery of a working product. Thereafter, HSP may or may not have clear ways of assigning responsibility for managing and governing the product’s use; in any case, HSU participation may be overlooked in these later stages in the life cycle of an eHealth resource. Participatory action research (PAR) [4] may be the launchpad for development [3], but at the end of development projects, there remains the need to operationalize and sustain the eHealth resources that have been created. The continuing quality assurance of eHealth resources within the auspicing health service also needs ongoing participation by HSU.

    Apart from operational needs for HSU participation, there are ethical reasons for it. HSP have an ethical responsibility for ensuring that eHealth innovations achieve health outcomes for HSU. HSP are committed to evidence-based practice, in this as in other aspects of their work. Therefore, when they think about developing and deploying new eHealth resources, where do they find what is recognized as good practice in HSU participation? There are so many case studies that it is a near-impossible task to synthesize them all; furthermore, some talk the talk but do not walk the walk of HSU participation, some do not follow any recognized methodology, and some finish early in the life cycle of the eHealth resource.

    There are numerous reviews and design guidelines that generalize about theories and methods of HSU participation in eHealth design. They emphasize the importance of the following basic principles:

    • appreciation and understanding from the outset, of the range of potential HSU characteristics, goals, needs, values, and perspectives on use [5-7]
    • attention to the needs of HSU not just as individual actors but also within their formal and informal care networks [8]
    • careful alignment of diverse concerns, attitudes, and perspectives that expert content creators, HSP, and HSU may have [9-12]
    • genuine active involvement so that HSU have an opportunity to identify practical problems and design, test, evaluate, and make decisions about technology in a range of environments [13-15]

    At the same time, they note that methods of HSU participation in eHealth design need to use human and other project resources judiciously. The themes are as follows:

    • ensuring that complex planning and evaluation models are able to be translated and streamlined to develop resources that are practical, feasible, and impactful in real-life settings [16]
    • taking a systematic approach to requirements specification to avoid mismatch with the organizational context and to support summative evaluation on a feature-specific level [17]
    • applying automation to expedite routine steps to create libraries of typical users and use cases and to manage unforeseen lessons learned for efficiency [18,19]
    Table 1. Glossary of terms.
    View this table

    Nevertheless, key considerations aside, it is difficult for HSP to identify from the literature a recognized, reliable methodological framework for engaging with HSU in the development of eHealth resources. A recent systematic review found that the literature variously encompassed 6 key phases and 17 different methods of participatory design, and it also found that sufficiency of reporting was poor and that no study undertook a robust assessment of efficacy [20]. This makes it difficult for HSP to study the effects of HSU participation in eHealth resources development on reach, adoption, acceptance, and efficacy of the intervention. Relative to other areas of health research, this type of study is immature, without widely endorsed methodological conventions for describing realistic aims for such projects or for determining valid measures of such effects [21].

    Therefore, this paper investigates reports of eHealth applications and tools and resource development to determine what methods have been used systematically to ensure full HSU participation. We sought to distill evidence of positive, negative, or other unanticipated effects that have arisen at any stage in the eHealth resource life cycle from various HSU participation methods. Within these participatory approaches, we identified the reported impact from the point of view of HSU and HSP.

    The impetus for this study began when the authors sought a strong research framework within which to undertake co-design of an eHealth initiative. The project was based on a print-based and workshop-based psychoeducational intervention called the Optimal Health Program (OHP). The authors wanted to ensure that they chose a rigorous methodological framework for redevelopment of OHP as an eHealth resource. Utilizing proven participatory methods would (1) optimize HSU engagement with the OHP resource that was developed, (2) strengthen the relevance of the resource to intended HSU, and (3) provide a logical foundation for long-term evaluation and improvement of the resource.

    Objectives

    This paper reviews published research reports that include detailed descriptions of participatory methods to engage HSU in eHealth resource development projects. Through synthesizing answers to the following questions, the objective of this paper is to support critical evaluation of this type of methodology and informed selection of appropriate approaches in future research and development projects:

    1. What types of eHealth resources have been developed using participatory processes, intended for what types of end users?
    2. What frameworks have been used from the very beginning of the design process to ensure participation by the intended end users in the development of eHealth resources?
    3. What methods within these frameworks have been most effective in supporting full involvement by intended end users of eHealth resources?
    4. What aspects of the participatory methods in these eHealth projects have emerged as being most important to end users?
    5. What positive, negative, or other unanticipated effects of participatory methods have the researchers reported at eHealth resource design, development, implementation, or evaluation stages?

    Methods

    Protocol and Registration

    This systematic review has been carried out in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [22,23]. Protocol CRD42017053838 was lodged with the PROSPERO international prospective register of systematic reviews.

    Information Sources

    A total of 9 bibliographic databases were searched, including 6 health and biomedical databases and 3 technology databases:

    • Ovid MEDLINE(R) Epub Ahead of Print, In-Process & Other Non-Indexed Citations, Ovid MEDLINE(R) Daily and Ovid MEDLINE(R) 1946 to Present (“MEDLINE”)
    • EMBASE (Embase.com ) (“EMBASE”)
    • CINAHL Plus with Full Text (EBSCOhost) (“CINAHL”)
    • PsycINFO 1806 to January Week 4 2017 (Ovid) (“PsycINFO”)
    • Ovid Emcare 1995 to 2016 week 49 (“Emcare”)
    • Cochrane Library, including Cochrane Database of Systematic Reviews; Database of Abstracts of Reviews of Effect; Cochrane Central Register of Controlled Trials; Cochrane Methodology Register; Health Technology Assessment Database; NHS Economic Evaluation Database; About the Cochrane Collaboration (“Cochrane”)
    • Web of Science Core Collection (“Web of Science”)
    • ACM Guide to Computing Literature (“ACM”)
    • IEEE Xplore Digital Library (“IEEE”)

    Additional articles were identified from reference lists of key articles and cited by references in Google Scholar.

    Search

    Search strategies were developed by an experienced medical research librarian (HW) in consultation with the OHP project leader (GM) and an expert eHealth researcher (KG).

    In December 2015, scoping searches were developed and run in MEDLINE, EMBASE, CINAHL, PsycINFO, and Cochrane. In April 2016, brief confirmatory searches were run in Google Scholar to consider gaps in the initial strategy and additional search terms or databases that could be included. As a result, search strategies were refined and rerun in the initial health and biomedical databases as well as 3 additional technology databases: Web of Science, ACM, and IEEE. In February 2017, searches were updated to include results to the end of 2016. At this stage, an additional health database, Emcare, was also searched.

    Within the health and biomedical databases (MEDLINE, EMBASE, CINAHL, PsycINFO, Emcare, and Cochrane) the search strategies combined the general concepts of user participation AND electronic resources AND program design. These search strategies were not limited to health-related conditions or resources because they yielded a small proportion of nonhealth-related results that could be removed manually. This enabled a very wide range of health conditions, HSU, organizations, and resources to be included in the results.

    Within the broader technology databases that are not health specific (Web of Science, ACM, and IEEE), the search strategies were necessarily limited to health-related resources, combining the general concepts of user participation AND electronic resources AND (health OR well-being) AND program design.

    A detailed search strategy was developed for MEDLINE using a combination of Medical Subject Headings (MeSH) and text words (Textbox 1). This was then adapted for the other databases, taking into account relevant subject headings and syntax. Search results were limited to publications dated from January 2006 to December 2016 and publications in English language. All database searches were updated in February 2017. Final search strategies for all databases are provided in Multimedia Appendix 1.


    Textbox 1. Search strategy for Ovid MEDLINE.
    View this box

    Study Selection

    The search results were exported from all bibliographic databases to Endnote bibliographic management software. Duplicates were identified and manually removed within Endnote by HW. The records were initially screened within Endnote on title and abstract by HW, excluding results that were clearly irrelevant, that is, not electronic technology, not health or well-being related, not development processes, or not involving end users. All potentially eligible records were exported from Endnote to Covidence, an online platform for managing the systematic review process. Covidence was used to screen records on title and abstract by any 2 of GM, HW, KG, and 1 additional reviewer using predefined inclusion and exclusion criteria as shown in Textboxes 2 and 3. All types of study design were eligible for initial inclusion.

    Full-text articles were obtained and uploaded to Covidence for all the available records that had been included based on title and abstract. When a number of articles reported on the same project, they were grouped into a single study to be reviewed together. The full text was reviewed independently by 2 reviewers, GM and HW, using additional inclusion and exclusion criteria (Textboxes 4 and 5).


    Textbox 2. Inclusion criteria for screening on title and abstract.
    View this box

    Textbox 3. Exclusion criteria for screening on title and abstract.
    View this box

    Textbox 4. Additional inclusion criteria for full-text review.
    View this box

    Textbox 5. Additional exclusion criteria for full-text review.
    View this box

    Data Collection Process

    The included studies were found to use qualitative, quantitative, and mixed methods for HSU participation in eHealth resource development; therefore, the Mixed Methods Appraisal Tool (MMAT) Version 2011 [24] was selected to analyze the rigor of these studies. The MMAT’s 19-assessment criteria were added to the extraction stage of Covidence. Each of the included full-text studies was assessed for methodological quality and rated according to the relevant MMAT criteria.

    The detail of MMAT ratings under each criterion was extracted and recorded in an Excel spreadsheet. MMAT scoring metrics were used to calculate a total score for each study in order to develop a hierarchy of evidence for the strength of different methodologies (Multimedia Appendix 2).

    The 2 reviewers, GM and HW, worked independently using MMAT to assess the methodological quality of papers and minimize risk of bias in assessing the literature. MMAT ratings and reasoning were compared, and conflicts were resolved through discussions between them.

    Risk of Bias in Individual Studies

    MMAT scores are typically 100%, 75%, 50%, and 25%. They work on the principle that a mixed-methods study is only as strong as its weakest part. This means that mixed-methods studies that have more criteria to meet (4 qualitative plus 4 quantitative plus 3 mixed method, equaling 11 criteria) could potentially be marked down more easily than studies that are purely qualitative and have fewer criteria to score (4 qualitative criteria only). In order to address this potential bias among study types, a decision was made to include an additional score of 90% to rationalize the difference that occurred between 100% and 75% in mixed-methods studies (Multimedia Appendix 2).

    After assessment, studies were grouped by MMAT score and sorted into alphabetical order according to the surname of the first author. Although study numbers were initially used by the reviewers for identification purposes, these have been removed so that there is no confusion about study number and ranking. All studies with the same MMAT score hold equal ranking.

    Data Items

    The 90 studies assessed according to MMAT are summarized descriptively in a table (Multimedia Appendix 3). First, the data items described in Table 2 were manually extracted from the full text by HW and recorded in Excel for analysis. These results were grouped, tallied, and exported into separate tables according to characteristics of the research scope, such as health area, technology, population, or country of study (Multimedia Appendices 4-7).

    Additional descriptive data were extracted from the full text of a subset of included studies, namely 30 studies that scored 90% or higher on MMAT. Data were extracted by HW and GM from the full text of each study using the data items listed in Table 3. These details were grouped, sorted, tallied, and exported into tables that summarize the main methods used to engage HSU in participatory development of eHealth resources.

    Methods, frameworks, and processes varied enormously among studies; therefore, a decision was made to allocate all reported methods to the 5 stages of a single framework in order to standardize comparison. The Center for eHealth Research and Disease Management (CeHRes) Roadmap [25] was chosen for this purpose because it was specific to eHealth, highly cited (approximately 400 times between 2011 and 2017), based on the review of many eHealth and development frameworks, process oriented (not just a list of methods but a focus on specific steps), and defined within 5 stages.

    Models and theories, participatory frameworks and interventions were extracted from the top 30 studies, and HW subsequently searched for additional mentions of them across the full text of the 90 MMAT–rated studies within Endnote.

    Risk of Bias Across Studies

    To minimize journal bias, a wide range of bibliographic databases were searched, including those with either a health focus or a technology focus. The search results were limited to English language, which could have created a cultural bias in the studies, although the 90 studies included in the quantitative analysis took place across 21 countries.

    Table 2. Data items extracted from 90 studies.
    View this table
    Table 3. Additional data items extracted from the top 30 studies.
    View this table

    The development of eHealth resources is a long process, sometimes taking many years, and many publications only reported a portion of the process, with only a few reporting the entire project up to final evaluation. As conference abstracts and grey literature were excluded in favor of journal articles, sections of the development process may have been reported elsewhere but not included in our evaluation. Reference lists and cited by references in Google Scholar were searched with respect to the top 90 studies to locate connected publications reporting later stages of development, but it is possible that some publications were either missed or published after our review timeframe.


    Results

    Database searches retrieved 24,674 records, which were exported to Endnote. Duplicates were removed by HW, leaving 15,117 records. These records were screened for broad relevance on title and abstract by HW and 13,096 records were excluded as clearly irrelevant. The remaining 2021 records were assessed for eligibility on title and abstract using the inclusion and exclusion criteria in Textboxes 2 and 3, and 1391 records were excluded.

    The 630 remaining records were combined into 603 studies, some of which involved multiple publications. All 603 studies were assessed for eligibility on full text, and 513 studies were excluded according to the criteria in Textboxes 4 and 5, leaving 90 studies for quantitative analysis. During the screening and full text review process, 12 additional records relating to the 90 studies were identified from reference lists or contact with authors, and those records were combined into the studies. See Figure 1 for the PRISMA flow diagram.

    A total of 90 studies were assessed for quality according to MMAT. Results are summarized in Table 4 and detailed results are available in Multimedia Appendix 2. An MMAT score of 100% was awarded to 28 studies and 2 studies scored 90%.

    Results From 90 Studies Included in Quantitative Analysis

    The 8 data items described in Table 2 were extracted from each of the 90 studies (Multimedia Appendix 3).

    The major health focus of each study was grouped into a hierarchy of 18 wider MeSH subject headings, summarized in Multimedia Appendix 4. The top 5 health areas were neoplasms (cancer), mental disorders, nutritional and metabolic diseases (including weight management), virus diseases (including HIV), cardiovascular diseases, and endocrine system diseases (including diabetes).

    Nine types of technology were reported in the 90 studies, and these are summarized in Multimedia Appendix 5. Websites (56 studies) and mobile apps (32 studies) were the main eHealth technologies developed. Other types of technology reported were decision tools, handheld computers, kiosk applications, personal health records, serious games, wearable devices, and telemonitoring.

    Studies targeting specific populations are summarized in Multimedia Appendix 6. Of the 90 studies, 22 (24%) were youth specific, and 9 (10%) focused on the aged. Of the 90 studies, 11 (12%) reported eHealth projects for women only, and 4 (4%) were for men only. Moreover, 3 studies (3%) had a Lesbian, Gay, Bisexual, Transgender, Queer or Questioning, and Intersex+ focus. Fourteen studies (16%) had either a cultural or multicultural focus, such as a bilingual app for Indigenous Australians or the development of a website in both France and Finland.

    The 90 studies took place in 21 countries, summarized in Multimedia Appendix 7. The top 6 countries were United States (33 studies), United Kingdom (15 studies), Netherlands (13 studies), Canada (7 studies), Sweden (6 studies), and Australia (6 studies). Studies also took place in Austria, Belgium, Czech Republic, Greece, Denmark, Finland, France, India, Spain, Ireland, Italy, New Zealand, Norway, Republic of Korea, and Saudi Arabia.

    Results From 30 Studies Included in Qualitative Synthesis

    The 30 studies scoring 90% or higher on MMAT were recorded in Excel spreadsheets and reviewed in detail. Data items listed in Table 3 were extracted for each study (Multimedia Appendix 8).

    The 30 studies are listed in Table 5, along with an indication of the CeHRes Roadmap stages reported. There was often a perceived overlap between stages 1 (contextual inquiry) and 2 (value specification) such as when focus groups may have covered both stages at once. Where this appeared to happen, it was reported in the spreadsheet and included in both stages in Table 5. Where the CeHRes Roadmap was particularly useful was in highlighting stages that were often not reported, such as operationalization or summative evaluation (Table 5). It is possible that some of these studies did address each stage but did not report them in journal articles that were reviewed.

    A summary of the 30 highest MMAT–rated studies is represented in Table 6 with details of the product developed, technology used and targeted population. The health area and general purpose of each eHealth project, categorized using Medical Subject Headings (MeSH) is summarized in Multimedia Appendix 9.

    The methods were recorded in Excel spreadsheets using the original terminology reported in each study. The details included the number of HSU or HSP involved in each process, the order of each activity as reported, and subprocesses within each method (for example, the type of design activity or workshop activity). These details are included in Multimedia Appendix 8. These detailed methods were then grouped so that they could be summarized using a consistent terminology and then compared. This summary of methods is included for each study in Table 7.

    Models and theories referred to in the top 30 studies are shown in Table 8.

    Tables 9-13 give an overview of the options used to satisfy each stage of the CeHRes roadmap and the popularity of these methods. Many of the methods reported may demonstrate formative evaluation processes occurring as part of an iterative process. We recommend referring to Multimedia Appendix 8 and the original references for additional information that may be able to identify the practical steps that were implemented.

    Figure 1. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram.
    View this figure
    Table 4. Summary of scoring of 90 studies according to the Mixed Methods Appraisal Tool Version 2011.
    View this table

    Lastly, the models and theories, participatory frameworks and interventions identified in the top 30 studies were searched across the full text of the 90 MMAT–rated studies within Endnote, and the results were ranked in order of prevalence in Multimedia Appendices 10-12.

    Twenty three models or theories were identified from the 30 studies scoring 90% or higher on MMAT as playing a role in the development of eHealth resources. The most often reported models and theories were Social Cognitive Theory (n=4, 13%) [144], Theory of Planned Behavior (n=3, 10%) [145], Transtheoretical Model (Prochaska Stages of Change) (n=3, 10%) [146], and the Persuasive Technology Theory/Behavior Model for Persuasive Design (n=3, 10%) [147]. A large variety of other models and theories were referred to, with little overlap between studies. Full results are recorded in Multimedia Appendix 10.

    A total of 24 named participatory frameworks or approaches were identified from the 30 studies scoring 90% or higher on MMAT (Multimedia Appendix 11). Only 20 of the 30 top scoring studies referred to a specific framework, with many studies referring more broadly to using participatory design or iterative design methods without reference to a particular named framework. The most often reported participatory frameworks or approaches were User-Centered Design ([UCD], n=5, 17%) [148], PAR framework (n=4, 13%) [149], CeHRes Roadmap (n=3, 10%) [25], Medical Research Council (MRC) Guide to Developing and Evaluating Complex Interventions (n=2, 7%) [150-152], and International Patient Decision Aid Standards Collaboration (n=2, 7%) [153].

    Some studies referred to specific interventions that were integral to the function of the eHealth resource that was developed. Key interventions identified in Multimedia Appendix 12 were Cognitive Behavior Therapy (CBT), Behavior Change Techniques, and Mindfulness.

    Table 5. Stages of the CeHRes Roadmap addressed in top 30 studies.
    View this table
    Table 6. Product, technology, and population in top 30 studies.
    View this table
    Table 7. Participatory frameworks and summary of methods in top 30 studies.
    View this table
    Table 8. Model or theory base in top 30 studies.
    View this table
    Table 9. Methods used in stage 1 (contextual enquiry) of the CeHRes Roadmap—top 30 studies.
    View this table
    Table 10. Methods used in stage 2 (value specification) of the CeHRes Roadmap—top 30 studies.
    View this table
    Table 11. Methods used in Stage 3 (design) of the CeHRes Roadmap—top 30 studies.
    View this table
    Table 12. Methods used in stage 4 (operationalization) of CeHRes Roadmap—top 30 studies.
    View this table
    Table 13. Methods used in stage 5 (summative evaluation) of CeHRes Roadmap—top 30 studies.
    View this table

    Discussion

    Overview

    In the era of digital health, we have a plethora of literature describing the need for better engagement with HSU to improve health care and health services, and we have access to the technologies to create a broad array of websites and mobile apps, but we lack detailed protocols for designing eHealth resources. This systematic review explored the participatory methods and frameworks used to engage HSU in the development of eHealth resources throughout the design process. UCD was most commonly reported but varied in its application and intention. Participatory methods promoting HSU engagement ranged from brief consultation via a review process to genuine collaboration, which included additional responsibility for the HSU in the actual creation process. Research and development projects that describe a conceptual model (such as Social Cognitive Theory) and a structured framework (such as the CeHRes Roadmap, which includes a diagram/flowchart) lay the foundations for us to gain greater insight into how particular processes lead to efficacious and effective eHealth resources.

    Electronic Health Initiatives Developed and the Characteristics of Health Service Users

    There have been extensive eHealth initiatives to address the issues of accessibility, engagement, health literacy, data collection, health promotion, early intervention, motivation, and behavioral change. Of the 90 MMAT-scored studies, websites and mobile apps make up the majority of eHealth initiatives presented in this review (Multimedia Appendix 5) with a strong multicultural focus (Multimedia Appendix 6). The end users of these eHealth initiatives were young adults, women, and the elderly (Multimedia Appendix 6) with the focus on cancer and mental health (Multimedia Appendix 4). The app has become an engagement tool used by HSP to make health information and health planning more interactive, interesting, and fun for HSU [30,32,35,36,42,43,56]. Moreover, participatory design is thought to enable young people to be creative and have substantial input into the resource development [30].

    Participatory Frameworks

    Analyzing the procedural frameworks used in our included studies, we found that no 2 studies reported their processes in the same way. The frameworks governing consumer participation were varied with the most reported being UCD, PAR Framework, CeHRes Roadmap, and MRC Guide to Developing and Evaluating Complex Interventions (Multimedia Appendix 11). The methods implemented to seek the HSU perspectives were also varied with the most reported being focus groups, surveys, interviews, prototype/storyboards, think aloud, and literature search (Tables 9-13). Theories and models that influenced procedures most commonly included cognition, behaviors, processes of change, motivation, and empowerment (Multimedia Appendix 10).

    The diversity in eHealth initiatives supports creativity, and to ensure validity and strengthen eHealth research, there is a need to integrate a set of protocols for HSU participation and reporting guidelines [154] available via the Enhancing the QUAlity and Transparency Of health Research Network. This would not constrain methodological innovation and would allow a more effective meta-analysis and comparison of participatory development studies.

    Methods Used in the Development of Resources

    This review looked for evidence of sound methods for engaging HSU during the development of eHealth apps, tools, and resources. We found relatively few reports that described HSU participation throughout development (ie, from contextual enquiry to summative evaluation, Table 5). Furthermore, many of these reports did not provide adequate details according to mixed-methods appraisal standards. As shown in Multimedia Appendix 2, studies out of 603 full texts reviewed met all of our inclusion criteria and scored 90% or higher according to MMAT. This suggests that research training, funding, and dissemination agencies need to attach far greater importance to reports that describe methods more rigorously.

    Others have observed that “The diverse communities working in digital health—including government stakeholders, technologists, clinicians, implementers, network operators, researchers, donors—have lacked a mutually understandable language with which to assess and articulate functionality” [155]. Tables 9-13 illustrates how deeply this lack has affected the production of cohesive research evidence, that is, it is virtually impossible to map the semantic relationships among the methodology elements to inform the discourse about what forms of participatory eHealth design work and why. Many methods are generic to human computer interaction, some take a broad behavioral approach and some include methods of measuring health outcomes in the particular area of health where the intervention is directed. One possible view is that this illustrates a flourishing of innovation and creativity. Another is that this creates a minefield for research training and peer reviewing and may represent a considerable waste of research resources.

    Analyzing the conceptual bases for the methods used in the 30 studies scoring 90% or higher on MMAT, we found much variety with 23 different models or theories reported (Multimedia Appendix 10). The most commonly occurring theories were Social Cognitive Theory, Theory of Planned Behavior, Transtheoretical Model, Persuasive Technology Theory, and Health Behavior Theory. This finding offers a sound basis in evidence for future researchers who wish to follow these precedents. However, we note that research in this area has not been informed by other potentially relevant theories (for example, theories that may account better for healthcare consumers’ economic, emotional, or empowerment motives for engagement) [156].

    Effective Involvement of Health Service Users

    This review looked for evidence about the effectiveness of particular approaches in terms of supporting involvement by HSU. Winterling reported strategies implemented to address engagement with HSU, including 1-person central contact, established expectation of roles, compensation for time, reaching a common agreement, and HSU seen as experts on patient perspective [63-66].

    It is also possible to reflect on the richness of the findings generated by particular approaches. As shown in Multimedia Appendix 13, each study reported between 2 and 10 major thematic outputs. Reports with relatively concise outputs were Bengtsson [28,29] using participatory research design and O’Brien [51] describing an array of approaches. The most extensive review was reported by Fleisher [68] using the Ottawa Decision Support Framework and participatory design and Goldenberg [42,43] using 3 types of iterative qualitative research approaches. In assessing effectiveness this way, unknown factors may be in play, such as sophistication of the data collection procedures, analytical expertise of the researchers, editorial constraints on reporting results, and temporal pressures on publication.

    Important Aspects of Participatory Methods for Health Service Users

    There were consistent themes that represented HSU priorities in eHealth initiatives across the selected 30 studies represented in Multimedia Appendix 13. Access to relevant, simple, and clear health information was reported consistently across most of the studies highlighting the importance of this information to make informed decisions in a timely manner. A well-designed eHealth resource that includes a framework supporting HSU involvement can significantly impact health literacy for both HSU and HSP. HSU involvement with the development of an eHealth resource created a collaborative process that required transparency and respect as well as clear mediation processes [53-55].

    Being involved in the development of an eHealth resource created the opportunity for HSU to clarify the user perspective and support the relevance of the final product. Despite the variety of websites and apps, HSU reported the need for improved access to information, coordination of care, interactivity with information provided, culturally specific information, patient education, and self-management. HSU also acknowledged the importance of confidentiality and privacy when exchanging personal health information over electronic networks.

    Impact of Participatory Methods Reported by Researchers

    The researchers reported a number of key issues highlighting the importance of participatory methods in creating an eHealth resource that was relevant to HSU. In Multimedia Appendix 13, an outline of the research recommendations was documented for the selected 30 studies. Researchers reported on the importance of utilizing a participatory design, which included an iterative process that increased the responsiveness and relevance of their eHealth initiatives. Having the HSU perspective from the beginning was important as well as ensuring that the process was genuinely collaborative with all participants respected and acknowledged. Utilizing a health behavior theory in combination with a participatory design was noted to enhance the eHealth resource. The theory base acknowledges the importance of motivation, empowerment, and stages of change in supporting the engagement and utilization of the eHealth resource. It was also noted that the eHealth resource needed to be interesting, engaging, and in some instances include a game-playing element. Creating a more positive approach enabled the HSU to engage with serious and difficult health issues and explore options for improved health. Not only did the eHealth resource need to be interesting but it also importantly needed to be intuitive and simple to navigate.

    Heckman [45] reported that their eHealth initiative was guided by intervention development, assessment guidelines for behavioral therapy, and health communication programs with health literacy best-practice. Utilizing a participatory design appeared to improve the relevance of the eHealth resource by addressing issues of culture, gender, age, and sexuality (Multimedia Appendix 6). Goldenberg [42,43] reported personalization along with interactive functionality promoted ownership for HSU. A majority of projects included both HSU and HSP in participatory methods across different developmental stages from contextual inquiry to summative evaluation of the project [28,29,31,34-38,40-43,45-55,58-61,63-67,69]. Evaluation is an integral part of participatory methodology; however, this was reported inconsistently across the 30 studies (Multimedia Appendix 8). The inclusion of a standardized tool to evaluate processes and outcomes from the HSU perspective, as part of a participatory framework, may address the need to bring more objectivity to evaluating various studies.

    The demand on time and financial resources to implement a participatory design was noted by some researchers [27,30,42,43,57,68]. Availability of resources was an important consideration throughout the design process, which was often iterative. With the rapid change in technology, there is an increasing demand for HSP to be agile and develop eHealth resources more quickly but still maintaining an evidence-based, best-practice approach inclusive of HSU participation.

    Limitations

    A limitation of our final dataset is that because of the number of papers retrieved, we decided to limit our analysis to published journal articles and to leave out full papers in conference proceedings. It is possible that there are strong participatory processes that have not been reported in detail, or at all, in the journal literature. Moreover, we did not include studies published in languages other than English and therefore we cannot be certain that our dataset reflects work being done around the world.

    As our focus was on the inclusion of HSU from the early development process onward, some studies were included that did not extend to a final evaluation of the product, and it was not always possible to consider the success or otherwise of the final eHealth product. As a part of our inclusion criteria, we required some evidence that a specific eHealth product was ultimately created or likely to be taken to completion.

    A limitation of our data analysis is that MMAT is a critical appraisal tool to assess the methodological quality of studies. It does not assess the quality of the writing or the content of the research; therefore, it is possible that we have overlooked papers that may be of high quality in other respects but which we have not rated highly here because of the way their methods sections are presented. For example, under MMAT, a paper will not score highly if it does not discuss the impact of the research or report the limitations of a mixed-methods study. The studies may not have rated highly under MMAT if they used both qualitative and quantitative methods but did not acknowledge that this constituted a mixed-methods study or if only selected aspects were reported. For example, a study that reported HSU participation only at the summative evaluation stage may have involved HSU earlier as well, but this would not register in our search results because we looked for descriptions of methods for HSU participation from initial design stages.

    Although categorizing all reported methods in these studies according to the 5 stages of the CeHRes Roadmap [25] was a generally useful way to compare processes across studies, absolute consistency was not achievable because of the wide variety of structuring reports, the differing terminology and naming conventions used for similar methods, and the difficulty in allocating all methods accurately to a particular process stage.

    Conclusions

    Agility of eHealth development is problematic in comparison to nonmedical industries as we seek to ensure safety and quality of care for HSU. It is a challenge for eHealth development to follow rigorous methods within a timeframe that responds to current needs, limited resources, and rapid technological changes. Methodological approaches to developing eHealth resources vary but the importance of engaging HSU in participatory design is consistently emphasized. By synthesizing the existing evidence about strong mixed methods for participatory development of eHealth resources, we anticipate that this systematic review will provide others with clearer guidance to plan more rapid and better-structured work of this kind.

    Acknowledgments

    The authors wish to thank Mark Merolli, Honorary Fellow Health and Biomedical Informatics Center, The University of Melbourne for his expertise and assistance. The authors also wish to thank Belinda Muscat and Zali Annersley, students and honorary St Vincent’s Hospital members for their assistance. This study was supported in part by a University of Melbourne Engagement grant.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Search strategies for all databases.

    PDF File (Adobe PDF File), 239KB

    Multimedia Appendix 2

    Scoring of 90 studies according to the Mixed Methods Appraisal Tool Version 2011.

    PDF File (Adobe PDF File), 471KB

    Multimedia Appendix 3

    Descriptive summary of 90 studies.

    PDF File (Adobe PDF File), 320KB

    Multimedia Appendix 4

    Major health focus in 90 studies.

    PDF File (Adobe PDF File), 186KB

    Multimedia Appendix 5

    Electronic health technology developed in 90 studies.

    PDF File (Adobe PDF File), 179KB

    Multimedia Appendix 6

    Targeted populations in 90 studies.

    PDF File (Adobe PDF File), 176KB

    Multimedia Appendix 7

    Countries where study took place in 90 studies.

    PDF File (Adobe PDF File), 183KB

    Multimedia Appendix 8

    Detail of data extracted from 30 studies.

    PDF File (Adobe PDF File), 264KB

    Multimedia Appendix 9

    Health area and purpose of product in top 30 studies categorised using Medical Subject Headings (MeSH).

    PDF File (Adobe PDF File), 195KB

    Multimedia Appendix 10

    Model or theory base in top 30 studies.

    PDF File (Adobe PDF File), 202KB

    Multimedia Appendix 11

    Participatory frameworks and approaches in top 30 studies.

    PDF File (Adobe PDF File), 201KB

    Multimedia Appendix 12

    Interventions in top 30 studies.

    PDF File (Adobe PDF File), 184KB

    Multimedia Appendix 13

    Themes and recommendations in top 30 studies.

    PDF File (Adobe PDF File), 435KB

    References

    1. World Health Organization. Constitution of the World Health Organization   URL: http://www.who.int/governance/eb/who_constitution_en.pdf [accessed 2018-06-20] [WebCite Cache]
    2. Okun S, Caligtan C. The engaged ePatient. In: Nelson R, Staggers N, editors. Health Informatics: An Interprofessional Approach. St Louis, Missouri: Elsevier; 2017:204-219.
    3. World Health Organization. 2016. Monitoring and evaluating digital health interventions: a practical guide to conducting research and assessment   URL: http://apps.who.int/iris/bitstream/10665/252183/1/9789241511766-eng.pdf [accessed 2019-01-17] [WebCite Cache]
    4. Loewenson R, Laurell AC, Hogstedt C, D'Ambruoso L, Shroff Z. Participatory Action Research in Health Systems: A Methods Reader. Harare, Zimbabwe: EQUINET; 2014.
    5. Asan O, Tyszka J, Fletcher KE. Capturing the patients' voices: planning for patient-centered electronic health record use. Int J Med Inform 2016 Dec;95:1-7 [FREE Full text] [CrossRef] [Medline]
    6. Gausepohl KA, Winchester WW, Smith-Jackson TL, Kleiner BM, Arthur JD. A conceptual model for the role of storytelling in design: leveraging narrative inquiry in user-centered design (UCD). Health Technol 2016 Jan 6;6(2):125-136. [CrossRef]
    7. Toraldo ML, Mangia G, Consiglio S, Mercurio R. A model for value creation: key role of the users as co-producers of e-health services. In: Proceedings IFKAD 2013. 2013 Presented at: International Forum on Knowledge Asset Dynamics; June 12-14, 2013; Zagreb, Croatia p. 1057-1070.
    8. Wherton J, Sugarhood P, Procter R, Hinder S, Greenhalgh T. Co-production in practice: how people with assisted living needs can help design and evolve technologies and services. Implement Sci 2015 May 26;10:75 [FREE Full text] [CrossRef] [Medline]
    9. Anderson JA, Godwin KM, Saleem JJ, Russell S, Robinson JJ, Kimmel B. Accessibility, usability, and usefulness of a web-based clinical decision support tool to enhance provider-patient communication around self-management to prevent (STOP) stroke. Health Informatics J 2014 Dec;20(4):261-274. [CrossRef] [Medline]
    10. Banna S, Hasan H, Dawson P. Understanding the diversity of user requirements for interactive online health services. Int J Healthcare Technol Manag 2016;15(3):253-271. [CrossRef]
    11. Crossley C, Fanfarelli J, McDaniel R. User experience design considerations for healthcare games and applications. In: 2016 IEEE International Conference on Serious Games and Applications for Health (SeGAH). 2016 Presented at: IEEE International Conference on Serious Games and Applications for Health; May 11-13, 2016; Orlando, Florida, USA p. 11-13. [CrossRef]
    12. Palm E, Nordgren A, Verweij M, Collste G. Ethically sound technology? Guidelines for interactive ethical assessment of personal health monitoring. Stud Health Technol Inform 2013;187:105-114. [CrossRef] [Medline]
    13. Clemensen J, Rothmann MJ, Smith AC, Caffery LJ, Danbjorg DB. Participatory design methods in telemedicine research. J Telemed Telecare 2016 Jan 1;23(9):780-785. [CrossRef] [Medline]
    14. Kushniruk A, Nøhr C. Participatory design, user involvement and health IT evaluation. In: Ammenwerth E, Rigby M, editors. Evidence-Based Health Informatics: Promoting Safety and Efficiency through Scientific Methods and Ethical Policy. Amsterdam: IOS Press; 2016:139-151.
    15. Park J, Kim JH, Park EJ, Ham SM. Analyzing user experience design of mobile hospital applications using the evaluation grid method. Wirel Pers Commun 2016 Dec;91(4):1591-1602. [CrossRef]
    16. Smits R, Bryant J, Sanson-Fisher R, Tzelepis F, Henskens F, Paul C, et al. Tailored and integrated web-based tools for improving psychosocial outcomes of cancer patients: the DoTTI development framework. J Med Internet Res 2014 Mar 14;16(3):11-23 [FREE Full text] [CrossRef] [Medline]
    17. van Velsen L, Wentzel J, van Gemert-Pijnen JE. Designing eHealth that matters via a multidisciplinary requirements development approach. JMIR Res Protoc 2013;2(1):e21 [FREE Full text] [CrossRef] [Medline]
    18. Hagedorn TJ, Krishnamurty S, Grosse IR. An information model to support user-centered design of medical devices. J Biomed Inform 2016 Dec;62:181-194 [FREE Full text] [CrossRef] [Medline]
    19. LeRouge C, Ma J, Sneha S, Tolle K. User profiles and personas in the design and development of consumer health technologies. Int J Med Inform 2013 Nov;82(11):e251-e268. [CrossRef] [Medline]
    20. Eyles H, Jull A, Dobson R, Firestone R, Whittaker R, te Morenga L, et al. Co-design of mHealth delivered interventions: a systematic review to assess key methods and processes. Curr Nutr Rep 2016 Jul 4;5(3):160-167. [CrossRef]
    21. DeSmet A, Thompson D, Baranowski T, Palmeira A, Verloigne M, de Bourdeaudhuij I. Is participatory design associated with the effectiveness of serious digital games for healthy lifestyle promotion? A meta-analysis. J Med Internet Res 2016 Apr 29;18(4):e94 [FREE Full text] [CrossRef] [Medline]
    22. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Ann Intern Med 2009 Aug 18;151(4):W65-W94. [CrossRef] [Medline]
    23. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009 Jul 21;6(7):e1000097 [FREE Full text] [CrossRef] [Medline]
    24. Pluye P, Robert E, Cargo M, Bartlett G, O'Cathain A, Griffiths F, et al. Mixed Methods Appraisal Tool (MMAT) – Version 2011. Montreal, Canada: Department of Family Medicine, McGill University; 2011.   URL: http:/​/mixedmethodsappraisaltoolpublic.​pbworks.com/​w/​file/​fetch/​84371689/​MMAT%202011%20criteria%20and%20tutorial%202011-06-29updated2014.​08.​21.​pdf [accessed 2019-01-17] [WebCite Cache]
    25. van Gemert-Pijnen JE, Nijland N, van Limburg M, Ossebaard HC, Kelders SM, Eysenbach G, et al. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res 2011;13(4):e111 [FREE Full text] [CrossRef] [Medline]
    26. Ahtinen A, Isomursu M, Ramiah S, Blom J. Advise, acknowledge, grow and engage: design principles for a mobile wellness application to support physical activity. Int J Hum Comput Int 2013;5(4):20-55. [CrossRef]
    27. Antypas K, Wangberg SC. Combining users' needs with health behavior models in designing an internet- and mobile-based intervention for physical activity in cardiac rehabilitation. JMIR Res Protoc 2014;3(1):e4 [FREE Full text] [CrossRef] [Medline]
    28. Bengtsson U, Kasperowski D, Ring L, Kjellgren K. Developing an interactive mobile phone self-report system for self-management of hypertension. Part 1: patient and professional perspectives. Blood Press 2014 Oct;23(5):288-295 [FREE Full text] [CrossRef] [Medline]
    29. Bengtsson U, Kjellgren K, Höfer S, Taft C, Ring L. Developing an interactive mobile phone self-report system for self-management of hypertension. Part 2: content validity and usability. Blood Press 2014 Oct;23(5):296-306 [FREE Full text] [CrossRef] [Medline]
    30. Buccieri K, Molleson G. Empowering homeless youth: building capacity through the development of mobile technology. J Community Pract 2015 May 18;23(2):238-254. [CrossRef]
    31. Clayman ML, Boberg EW, Makoul G. The use of patient and provider perspectives to develop a patient-oriented website for women diagnosed with breast cancer. Patient Educ Couns 2008 Sep;72(3):429-435. [CrossRef] [Medline]
    32. Cordova D, Bauermeister JA, Fessler K, Delva J, Nelson A, Nurenberg R, et al. A community-engaged approach to developing an mHealth HIV/STI and drug abuse preventive intervention for primary care: a qualitative study. JMIR Mhealth Uhealth 2015;3(4):e106 [FREE Full text] [CrossRef] [Medline]
    33. Dabbs AD, Myers BA, McCurry KR, Dunbar-Jacob J, Hawkins RP, Begey A, et al. User-centered design and interactive health technologies for patients. Comput Inform Nurs 2009;27(3):175-183 [FREE Full text] [CrossRef] [Medline]
    34. Das A, Svanæs D. Human-centred methods in the design of an e-health solution for patients undergoing weight loss treatment. Int J Med Inform 2013 Nov;82(11):1075-1091. [CrossRef] [Medline]
    35. Davies J, Bukulatjpi S, Sharma S, Caldwell L, Johnston V, Davis JS. Development of a culturally appropriate bilingual electronic app about hepatitis B for indigenous Australians: towards shared understandings. JMIR Res Protoc 2015 Jun 10;4(2):e70 [FREE Full text] [CrossRef] [Medline]
    36. Davies J, Bukulatjpi S, Sharma S, Davis J, Johnston V. "Only your blood can tell the story"--a qualitative research study using semi-structured interviews to explore the hepatitis B related knowledge, perceptions and experiences of remote dwelling Indigenous Australians and their health care providers in northern Australia. BMC Public Health 2014 Nov 28;14:1233 [FREE Full text] [CrossRef] [Medline]
    37. Duman-Lubberding S, van Uden-Kraan CF, Jansen F, Witte BI, van der Velden LA, Lacko M, et al. Feasibility of an eHealth application "OncoKompas" to improve personalized survivorship cancer care. Support Care Cancer 2016 May;24(5):2163-2171 [FREE Full text] [CrossRef] [Medline]
    38. Duman-Lubberding S, van Uden-Kraan CF, Peek N, Cuijpers P, Leemans CR, Verdonck-de Leeuw IM. An eHealth application in head and neck cancer survivorship care: health care professionals' perspectives. J Med Internet Res 2015 Oct 21;17(10):e235 [FREE Full text] [CrossRef] [Medline]
    39. Fennell KM, Turnbull DA, Bidargaddi N, McWha JL, Davies M, Olver I. The consumer-driven development and acceptability testing of a website designed to connect rural cancer patients and their families, carers and health professionals with appropriate information and psychosocial support. Eur J Cancer Care (Engl) 2017 Sep;26(5). [CrossRef] [Medline]
    40. Fonda SJ, Kedziora RJ, Vigersky RA, Bursell S. Combining iGoogle and personal health records to create a prototype personal health application for diabetes self-management. Telemed J E Health 2010 May;16(4):480-489. [CrossRef] [Medline]
    41. Fonda SJ, Kedziora RJ, Vigersky RA, Bursell S. Evolution of a web-based, prototype personal health application for diabetes self-management. J Biomed Inform 2010 Oct;43(5 Suppl):S17-S21 [FREE Full text] [CrossRef] [Medline]
    42. Goldenberg T, McDougal SJ, Sullivan PS, Stekler JD, Stephenson R. Preferences for a mobile HIV prevention app for men who have sex with men. JMIR Mhealth Uhealth 2014;2(4):e47 [FREE Full text] [CrossRef] [Medline]
    43. Goldenberg T, McDougal SJ, Sullivan PS, Stekler JD, Stephenson R. Building a mobile HIV prevention app for men who have sex with men: an iterative and community-driven process. JMIR Public Health Surveill 2015;1(2):e18 [FREE Full text] [CrossRef] [Medline]
    44. Gunn K, Turnbull D, McWha JL, Davies M, Olver I. Psychosocial service use: a qualitative exploration from the perspective of rural Australian cancer patients. Support Care Cancer 2013 Sep;21(9):2547-2555. [CrossRef] [Medline]
    45. Heckman C, Darlow S, Munshi T, Caruso C, Ritterband L, Raivitch S, et al. Development of an internet intervention to address behaviors associated with skin cancer risk among young adults. Internet Interv 2015 Sep;2(3):340-350 [FREE Full text] [CrossRef] [Medline]
    46. Kelders SM, Pots WT, Oskam MJ, Bohlmeijer ET, van Gemert-Pijnen JE. Development of a web-based intervention for the indicated prevention of depression. BMC Med Inform Decis Mak 2013;13:26 [FREE Full text] [CrossRef] [Medline]
    47. Lubberding S, van Uden-Kraan CF, te Velde EA, Cuijpers P, Leemans CR, Verdonck-de Leeuw IM. Improving access to supportive cancer care through an eHealth application: a qualitative needs assessment among cancer survivors. J Clin Nurs 2015 May;24(9-10):1367-1379. [CrossRef] [Medline]
    48. Meyer D. Online self-help: developing a student-focused website for depression. Couns Psychother Res 2007 Sep;7(3):151-156. [CrossRef]
    49. Miller SM, Hudson SV, Hui SA, Diefenbach MA, Fleisher L, Raivitch S, et al. Development and preliminary testing of PROGRESS: a Web-based education program for prostate cancer survivors transitioning from active treatment. J Cancer Surviv 2015 Sep;9(3):541-553 [FREE Full text] [CrossRef] [Medline]
    50. Morrison D, Mair FS, Chaudhuri R, McGee-Lennon M, Thomas M, Thomson NC, et al. Details of development of the resource for adults with asthma in the RAISIN (randomized trial of an asthma internet self-management intervention) study. BMC Med Inform Decis Mak 2015 Jul 28;15:57 [FREE Full text] [CrossRef] [Medline]
    51. O'Brien N, Heaven B, Teal G, Evans EH, Cleland C, Moffatt S, et al. Integrating evidence from systematic reviews, qualitative research, and expert knowledge using co-design techniques to develop a web-based intervention for people in the retirement transition. J Med Internet Res 2016 Aug 3;18(8):e210 [FREE Full text] [CrossRef] [Medline]
    52. Peute LW, Knijnenburg SL, Kremer LC, Jaspers MW. A concise and practical framework for the development and usability evaluation of patient information websites. Appl Clin Inform 2015;6(2):383-399 [FREE Full text] [CrossRef] [Medline]
    53. Revenäs A, Martin C, Opova CH, Brusewitz M, Keller C, Åsenlöf P. A mobile internet service for self-management of physical activity in people with rheumatoid arthritis: challenges in advancing the co-design process during the requirements specification phase. JMIR Res Protoc 2015 Sep 17;4(3):e111 [FREE Full text] [CrossRef] [Medline]
    54. Revenäs A, Opava CH, Åsenlöf P. Lead users' ideas on core features to support physical activity in rheumatoid arthritis: a first step in the development of an internet service using participatory design. BMC Med Inform Decis Mak 2014 Mar 22;14:21 [FREE Full text] [CrossRef] [Medline]
    55. Revenäs A, Opava CH, Martin C, Demmelmaier I, Keller C, Åsenlöf P. Development of a web-based and mobile app to support physical activity in individuals with rheumatoid arthritis: results from the second step of a co-design process. JMIR Res Protoc 2015;4(1):e22 [FREE Full text] [CrossRef] [Medline]
    56. Sandlund M, Lindgren H, Pohl P, Melander-Wikman A, Bergvall-Kareborn B, Lundin-Olsson L. Towards a mobile exercise application to prevent falls: a participatory design process. Int J Child Health Hum Dev 2016;9(3):389-398 [FREE Full text]
    57. Schnall R, Rojas M, Bakken S, Brown W, Carballo-Dieguez A, Carry M, et al. A user-centered model for designing consumer mobile health (mHealth) applications (apps). J Biomed Inform 2016 Apr;60:243-251. [CrossRef] [Medline]
    58. Skjøth MM, Hansen HP, Draborg E, Pedersen CD, Lamont RF, Jørgensen JS. Informed choice for participation in Down syndrome screening: development and content of a web-based decision aid. JMIR Res Protoc 2015;4(3):e113 [FREE Full text] [CrossRef] [Medline]
    59. Stinson JN, Lalloo C, Harris L, Isaac L, Campbell F, Brown S, et al. iCanCope with Pain™: user-centred design of a web- and mobile-based self-management program for youth with chronic pain based on identified health care needs. Pain Res Manag 2014;19(5):257-265 [FREE Full text] [Medline]
    60. van Bruinessen IR, van Weel-Baumgarten EM, Gouw H, Zijlstra JM, Albada A, van Dulmen S. Barriers and facilitators to effective communication experienced by patients with malignant lymphoma at all stages after diagnosis. Psychooncology 2013 Dec;22(12):2807-2814. [CrossRef] [Medline]
    61. van Bruinessen IR, van Weel-Baumgarten EM, Snippe HW, Gouw H, Zijlstra JM, van Dulmen S. Active patient participation in the development of an online intervention. JMIR Res Protoc 2014;3(4):e59 [FREE Full text] [CrossRef] [Medline]
    62. Widman L, Golin CE, Noar SM, Massey J, Prinstein MJ. ProjectHeartforGirls.com: development of a web-based HIV/STD prevention program for adolescent girls emphasizing sexual communication skills. AIDS Educ Prev 2016 Oct;28(5):365-377 [FREE Full text] [CrossRef] [Medline]
    63. Winterling J, Wiklander M, Obol CM, Lampic C, Eriksson LE, Pelters B, et al. Development of a self-help web-based intervention targeting young cancer patients with sexual problems and fertility distress in collaboration with patient research partners. JMIR Res Protoc 2016 Apr 12;5(2):e60 [FREE Full text] [CrossRef] [Medline]
    64. Armuand GM, Wettergren L, Rodriguez-Wallberg KA, Lampic C. Women more vulnerable than men when facing risk for treatment-induced infertility: a qualitative study of young adults newly diagnosed with cancer. Acta Oncol 2015 Feb;54(2):243-252. [CrossRef] [Medline]
    65. Jervaeus A, Nilsson J, Eriksson LE, Lampic C, Widmark C, Wettergren L. Exploring childhood cancer survivors' views about sex and sexual experiences-findings from online focus group discussions. Eur J Oncol Nurs 2016 Feb;20:165-172 [FREE Full text] [CrossRef] [Medline]
    66. Nilsson J, Jervaeus A, Lampic C, Eriksson LE, Widmark C, Armuand GM, et al. 'Will I be able to have a baby?' Results from online focus group discussions with childhood cancer survivors in Sweden. Hum Reprod 2014 Dec;29(12):2704-2711 [FREE Full text] [CrossRef] [Medline]
    67. Ennis L, Robotham D, Denis M, Pandit N, Newton D, Rose D, et al. Collaborative development of an electronic personal health record for people with severe and enduring mental health problems. BMC Psychiatry 2014;14(1):305 [FREE Full text] [CrossRef] [Medline]
    68. Fleisher L, Ruggieri DG, Miller SM, Manne S, Albrecht T, Buzaglo J, et al. Application of best practice approaches for designing decision support tools: the preparatory education about clinical trials (PRE-ACT) study. Patient Educ Couns 2014 Jul;96(1):63-71 [FREE Full text] [CrossRef] [Medline]
    69. Ennis L, Rose D, Denis M, Pandit N, Wykes T. Can't surf, won't surf: the digital divide in mental health. J Ment Health 2012 Aug;21(4):395-403 [FREE Full text] [CrossRef] [Medline]
    70. Alnasser A, Sathiaseelan A, Al-Khalifa A, Marais D. Development of 'Twazon': an Arabic app for weight loss. JMIR Res Protoc 2016 May 16;5(2):e76 [FREE Full text] [CrossRef] [Medline]
    71. Alnasser AA, Alkhalifa AS, Sathiaseelan A, Marais D. What overweight women want from a weight loss app: a qualitative study on arabic women. JMIR Mhealth Uhealth 2015;3(2):e41 [FREE Full text] [CrossRef] [Medline]
    72. Alvarado-Martel D, Cañas F, Velasco R, Alcubierre N, López-Ríos L, Rius F, et al. Design, construction, and implementation of an online platform for patients with type 1 diabetes: EncoDiab. Patient Prefer Adherence 2015;9:767-775 [FREE Full text] [CrossRef] [Medline]
    73. Andersen T, Bansler R, Kensing F, Moll J, Nielsen K. Alignment of Concerns: A Design Rationale for Patient Participation in eHealth. In: Proceedings of the 2014 47th Hawaii International Conference on System Sciences. 2014 Presented at: HICSS'14; January 6-9, 2014; Hawaii. [CrossRef]
    74. Andersen T, Bjørn P, Kensing F, Moll J. Designing for collaborative interpretation in telemonitoring: re-introducing patients as diagnostic agents. Int J Med Inform 2011 Aug;80(8):e112-e126. [CrossRef] [Medline]
    75. Armstrong N, Hearnshaw H, Powell J, Dale J. Stakeholder perspectives on the development of a virtual clinic for diabetes care: qualitative study. J Med Internet Res 2007 Aug 9;9(3):e23 [FREE Full text] [CrossRef] [Medline]
    76. Armstrong N, Powell J. Preliminary test of an internet-based diabetes self-management tool. J Telemed Telecare 2008;14(3):114-116. [CrossRef] [Medline]
    77. Armstrong N, Powell J. Patient perspectives on health advice posted on internet discussion boards: a qualitative study. Health Expect 2009 Sep;12(3):313-320. [CrossRef] [Medline]
    78. Atkinson NL, Saperstein SL, Desmond SM, Gold RS, Billing AS, Tian J. Rural eHealth nutrition education for limited-income families: an iterative and user-centered design approach. J Med Internet Res 2009 Jun 22;11(2):e21 [FREE Full text] [CrossRef] [Medline]
    79. Bae J, Wolpin S, Kim E, Lee S, Yoon S, An K. Development of a user-centered health information service system for depressive symptom management. Nurs Health Sci 2009 Jun;11(2):185-193. [CrossRef] [Medline]
    80. Bessell A, Clarke A, Harcourt D, Moss TP, Rumsey N. Incorporating user perspectives in the design of an online intervention tool for people with visible differences: face IT. Behav Cogn Psychother 2010 Oct;38(5):577-596. [CrossRef] [Medline]
    81. Boyd AD, Moores K, Shah V, Sadhu E, Shroff A, Groo V, et al. My Interventional Drug-Eluting Stent Educational App (MyIDEA): patient-centered design methodology. JMIR Mhealth Uhealth 2015;3(3):e74 [FREE Full text] [CrossRef] [Medline]
    82. Buckingham CD, Adams A, Vail L, Kumar A, Ahmed A, Whelan A, et al. Integrating service user and practitioner expertise within a web-based system for collaborative mental-health risk and safety management. Patient Educ Couns 2015 Oct;98(10):1189-1196. [CrossRef] [Medline]
    83. Cade J, Carter M, Wark P, Alwan N, Evans C, Albar S, et al. Development of a UK online 24-hour dietary recall assessment tool: 'My Food 24'. Eur J Epidemiol 2013;28(Supp 1):S50-S51. [CrossRef] [Medline]
    84. Cafazzo JA, Casselman M, Hamming N, Katzman DK, Palmert MR. Design of an mHealth app for the self-management of adolescent type 1 diabetes: a pilot study. J Med Internet Res 2012;14(3):e70 [FREE Full text] [CrossRef] [Medline]
    85. Carter MC, Albar SA, Morris MA, Mulla UZ, Hancock N, Evans CE, myfood24 Consortium. Development of a UK Online 24-h Dietary Assessment Tool: myfood24. Nutrients 2015 Jun;7(6):4016-4032 [FREE Full text] [CrossRef] [Medline]
    86. Cnossen IC, van Uden-Kraan CF, Eerenstein SE, Rinkel R, Aalders IJ, van den Berg K, et al. A participatory design approach to develop a web-based self-care program supporting early rehabilitation among patients after total laryngectomy. Folia Phoniatr Logop 2015;67(4):193-201 [FREE Full text] [CrossRef] [Medline]
    87. Colombo C, Filippini G, Synnot A, Hill S, Guglielmino R, Traversa S, et al. Development and assessment of a website presenting evidence-based information for people with multiple sclerosis: the IN-DEEP project. BMC Neurol 2016 Mar 2;16:30 [FREE Full text] [CrossRef] [Medline]
    88. Coyne I, Prizeman G, Sheehan A, Malone H, While AE. An e-health intervention to support the transition of young people with long-term illnesses to adult healthcare services: design and early use. Patient Educ Couns 2016 Dec;99(9):1496-1504. [CrossRef] [Medline]
    89. Danaher BG, Milgrom J, Seeley JR, Stuart S, Schembri C, Tyler MS, et al. Web-based intervention for postpartum depression: formative research and design of the MomMoodBooster Program. JMIR Res Protoc 2012;1(2):e18 [FREE Full text] [CrossRef] [Medline]
    90. Davies RJ, Nugent CD, Donnelly MP, Hettinga M, Meiland FJ, Moelaert F, et al. A user driven approach to develop a cognitive prosthetic to address the unmet needs of people with mild dementia. Pervasive Mob Comput 2009 Jun;5(3):253-267. [CrossRef]
    91. Dykes PC, Stade D, Chang F, Dalal A, Getty G, Kandala R, et al. Participatory design and development of a patient-centered toolkit to engage hospitalized patients and care partners in their plan of care. AMIA Annu Symp Proc 2014;2014:486-495 [FREE Full text] [Medline]
    92. Enah C, Moneyham L, Vance DE, Gakumo CA, Chandler M. Make it like the real world: adolescents’ recommendations for the design of a digital HIV prevention game. J HIV AIDS Soc Serv 2014 May 28;13(2):163-178. [CrossRef]
    93. Enah C, Piper K, Moneyham L. Qualitative evaluation of the relevance and acceptability of a web-based HIV prevention game for rural adolescents. J Pediatr Nurs 2015 Apr;30(2):321-328. [CrossRef] [Medline]
    94. Fink A, Beck JC. Developing and evaluating a website to guide older adults in their health information searches: A mixed-methods approach. J Appl Gerontol 2015 Aug;34(5):633-651. [CrossRef] [Medline]
    95. Fledderus M, Schreurs KM, Bohlmeijer ET, Vollenbroek-Hutten MM. Development and pilot evaluation of an online relapse-prevention program based on acceptance and commitment therapy for chronic pain patients. JMIR Hum Factors 2015 Jan 5;2(1):e1 [FREE Full text] [CrossRef] [Medline]
    96. Graham ML, Uesugi KH, Niederdeppe J, Gay GK, Olson CM. The theory, development, and implementation of an e-intervention to prevent excessive gestational weight gain: e-Moms Roc. Telemed J E Health 2014 Dec;20(12):1135-1142. [CrossRef] [Medline]
    97. Hearn L, Miller M, Fletcher A. Online healthy lifestyle support in the perinatal period: what do women want and do they use it? Aust J Prim Health 2013;19(4):313-318. [CrossRef] [Medline]
    98. Hearn L, Miller M, Lester L. Reaching perinatal women online: the Healthy You, Healthy Baby website and app. J Obes 2014;2014:573928 [FREE Full text] [CrossRef] [Medline]
    99. Hightow-Weidman LB, Fowler B, Kibe J, McCoy R, Pike E, Calabria M, et al. HealthMpowerment.org: development of a theory-based HIV/STI website for young black MSM. AIDS Educ Prev 2011 Feb;23(1):1-12 [FREE Full text] [CrossRef] [Medline]
    100. Horne B, Newsham A, Velikova G, Liebersbach S, Gilleece M, Wright P. Development and evaluation of a specifically designed website for haematopoietic stem cell transplant patients in Leeds. Eur J Cancer Care (Engl) 2016 May;25(3):402-418. [CrossRef] [Medline]
    101. Kattelmann KK, White AA, Greene GW, Byrd-Bredbenner C, Hoerr SL, Horacek TM, et al. Development of Young Adults Eating and Active for Health (YEAH) internet-based intervention via a community-based participatory research model. J Nutr Educ Behav 2014;46(2):S10-S25. [CrossRef] [Medline]
    102. Kim KK, Logan HC, Young E, Sabee CM. Youth-centered design and usage results of the iN Touch mobile self-management program for overweight/obesity. Pers Ubiquit Comput 2015 Jan;19(1):59-68. [CrossRef]
    103. Kuijpers W, Groen WG, Loos R, Oldenburg HS, Wouters MW, Aaronson NK, et al. An interactive portal to empower cancer survivors: a qualitative study on user expectations. Support Care Cancer 2015 Sep;23(9):2535-2542 [FREE Full text] [CrossRef] [Medline]
    104. Kuijpers W, Groen WG, Oldenburg HS, Wouters MW, Aaronson NK, van Harten WH. Development of MijnAVL, an interactive portal to empower breast and lung cancer survivors: an iterative, multi-stakeholder approach. JMIR Res Protoc 2015 Jan 22;4(1):e14 [FREE Full text] [CrossRef] [Medline]
    105. Lee MK, Park HA, Yun YH, Chang YJ. Development and formative evaluation of a web-based self-management exercise and diet intervention program with tailored motivation and action planning for cancer survivors. JMIR Res Protoc 2013;2(1):e11 [FREE Full text] [CrossRef] [Medline]
    106. Martin-Hammond AM, Abegaz T, Gilbert JE. Designing an over-the-counter consumer decision-making tool for older adults. J Biomed Inform 2015 Oct;57:113-123 [FREE Full text] [CrossRef] [Medline]
    107. Neville C, da Costa D, Mill C, Rochon M, Aviña-Zubieta JA, Pineau CA, et al. The needs of persons with lupus and health care providers: a qualitative study aimed toward the development of the Lupus Interactive Navigator™. Lupus 2014 Feb;23(2):176-182. [CrossRef] [Medline]
    108. Neville C, da Costa D, Rochon M, Peschken CA, Pineau CA, Bernatsky S, et al. Development of the Lupus Interactive Navigator as an empowering web-based eHealth tool to facilitate lupus management: users perspectives on usability and acceptability. JMIR Res Protoc 2016 May 30;5(2):e44 [FREE Full text] [CrossRef] [Medline]
    109. Schlosser D, Campellone T, Kim D, Truong B, Vergani S, Ward C, et al. Feasibility of PRIME: a cognitive neuroscience-informed mobile app intervention to enhance motivated behavior and improve quality of life in recent onset schizophrenia. JMIR Res Protoc 2016 Apr 28;5(2):e77 [FREE Full text] [CrossRef] [Medline]
    110. Siek KA, Khan DU, Ross SE, Haverhals LM, Meyers J, Cali SR. Designing a personal health application for older adults to manage medications: a comprehensive case study. J Med Syst 2011 Oct;35(5):1099-1121. [CrossRef] [Medline]
    111. Todd NJ, Jones SH, Lobban FA. What do service users with bipolar disorder want from a web-based self-management intervention? A qualitative focus group study. Clin Psychol Psychother 2013;20(6):531-543. [CrossRef] [Medline]
    112. Trudeau KJ, Ainscough JL, Trant M, Starker J, Cousineau TM. Identifying the educational needs of menopausal women: a feasibility study. Womens Health Issues 2011;21(2):145-152 [FREE Full text] [CrossRef] [Medline]
    113. Noordegraaf AV, Huirne JA, Pittens CA, van Mechelen W, Broerse JE, Brölmann HA, et al. eHealth program to empower patients in returning to normal activities and work after gynecological surgery: intervention mapping as a useful method for development. J Med Internet Res 2012 Oct 19;14(5):e124 [FREE Full text] [CrossRef] [Medline]
    114. Ward R, Taha KM. Patient involvement as experts in the development and assessment of a smartphone app as a patient education tool for the management of thalassemia and iron overload syndromes. Hemoglobin 2016 Sep;40(5):323-329. [CrossRef] [Medline]
    115. Waterlander W, Whittaker R, McRobbie H, Dorey E, Ball K, Maddison R, et al. Development of an evidence-based mHealth weight management program using a formative research process. JMIR Mhealth Uhealth 2014;2(3):e18 [FREE Full text] [CrossRef] [Medline]
    116. Whitehouse SR, Lam PY, Balka E, McLellan S, Deevska M, Penn D, et al. Co-creation with TickiT: designing and evaluating a clinical eHealth platform for youth. JMIR Res Protoc 2013 Oct 18;2(2):e42 [FREE Full text] [CrossRef] [Medline]
    117. Whittaker R, Maddison R, McRobbie H, Bullen C, Denny S, Dorey E, et al. A multimedia mobile phone-based youth smoking cessation intervention: findings from content development and piloting studies. J Med Internet Res 2008 Nov 25;10(5):e49 [FREE Full text] [CrossRef] [Medline]
    118. Williamson H, Griffiths C, Harcourt D. Developing young person's Face IT: Online psychosocial support for adolescents struggling with conditions or injuries affecting their appearance. Health Psychol Open 2015 Jul;2(2):2055102915619092 [FREE Full text] [CrossRef] [Medline]
    119. Williamson H, Hamlet C, White P, Marques EM, Cadogan J, Perera R, et al. Study protocol of the YP Face IT feasibility study: comparing an online psychosocial intervention versus treatment as usual for adolescents distressed by appearance-altering conditions/injuries. BMJ Open 2016 Dec 3;6(10):e012423 [FREE Full text] [CrossRef] [Medline]
    120. Wright CJ, Dietze PM, Crockett B, Lim MS. Participatory development of MIDY (Mobile Intervention for Drinking in Young people). BMC Public Health 2016;16:184 [FREE Full text] [CrossRef] [Medline]
    121. Wysocki T, Hirschfeld F, Miller L, Izenberg N, Dowshen SA, Taylor A, et al. Consideration of insulin pumps or continuous glucose monitors by adolescents with type 1 diabetes and their parents: stakeholder engagement in the design of web-based decision aids. Diabetes Educ 2016 Dec;42(4):395-407. [CrossRef] [Medline]
    122. Todd NJ, Jones SH, Hart A, Lobban FA. A web-based self-management intervention for bipolar disorder 'Living with Bipolar': a feasibility randomised controlled trial. J Affect Disord 2014 Dec;169:21-29. [CrossRef] [Medline]
    123. Meiland FJ, Bouman AI, Sävenstedt S, Bentvelzen S, Davies RJ, Mulvenna MD, et al. Usability of a new electronic assistive device for community-dwelling persons with mild dementia. Aging Ment Health 2012;16(5):584-591. [CrossRef] [Medline]
    124. Crosby LE, Ware RE, Goldstein A, Walton A, Joffe NE, Vogel C, et al. Development and evaluation of iManage: a self-management app co-designed by adolescents with sickle cell disease. Pediatr Blood Cancer 2017 Jan;64(1):139-145. [CrossRef] [Medline]
    125. Foster C, Calman L, Grimmett C, Breckons M, Cotterell P, Yardley L, et al. Managing fatigue after cancer treatment: development of RESTORE, a web-based resource to support self-management. Psychooncology 2015 Aug;24(8):940-949. [CrossRef] [Medline]
    126. Gordon EJ, Feinglass J, Carney P, Ramirez D, Olivero M, O'Connor K, et al. An interactive, bilingual, culturally targeted website about living kidney donation and transplantation for hispanics: development and formative evaluation. JMIR Res Protoc 2015 Apr 20;4(2):e42 [FREE Full text] [CrossRef] [Medline]
    127. Grant RW, Pandiscio JC, Pajolek H, Woulfe A, Pelletier A, Kvedar J, et al. Implementation of a web-based tool for patient medication self-management: the Medication Self-titration Evaluation Programme (Med-STEP) for blood pressure control. Inform Prim Care 2012;20(1):57-67 [FREE Full text] [CrossRef] [Medline]
    128. Groussard PY, Pigot H, Giroux S. From conception to evaluation of mobile services for people with head injury: a participatory design perspective. Neuropsychol Rehabil 2018 Jul;28(5):667-688. [CrossRef] [Medline]
    129. Hallett J, Maycock B, Kypri K, Howat P, McManus A. Development of a web-based alcohol intervention for university students: processes and challenges. Drug Alcohol Rev 2009 Jan;28(1):31-39. [CrossRef] [Medline]
    130. Heinrich E, Schaper N, de Vries N. Development of the web-based type 2 diabetes education programme: DIEP. Eur Diabetes Nurs 2009 Sep;6(2):51-56. [CrossRef]
    131. McCarthy O, Carswell K, Murray E, Free C, Stevenson F, Bailey JV. What young people want from a sexual health website: design and development of Sexunzipped. J Med Internet Res 2012;14(5):e127 [FREE Full text] [CrossRef] [Medline]
    132. McCrindle RJ, Williams VM, Victor CR, Harvey AP, Nyman SR, Barrett J, et al. Wearable device to assist independent living. Int J Disabil Hum Dev 2011;10(4):349-354. [CrossRef]
    133. Timmerman JG, Tönis TM, Dekker-van Weering MG, Stuiver MM, Wouters MW, van Harten WH, et al. Co-creation of an ICT-supported cancer rehabilitation application for resected lung cancer survivors: design and evaluation. BMC Health Serv Res 2016 Dec 27;16:155 [FREE Full text] [CrossRef] [Medline]
    134. Willems RA, Bolman CA, Mesters I, Kanera IM, Beaulen AA, Lechner L. The Kanker Nazorg Wijzer (Cancer Aftercare Guide) protocol: the systematic development of a web-based computer tailored intervention providing psychosocial and lifestyle support for cancer survivors. BMC Cancer 2015;15:580 [FREE Full text] [CrossRef] [Medline]
    135. Ashurst EJ, Jones RB, Abraham C, Jenner M, Boddy K, Besser RE, et al. The diabetes app challenge: user-led development and piloting of internet applications enabling young people with diabetes to set the focus for their diabetes consultations. Med 2.0 2014 Nov 7;3(2):e5 [FREE Full text] [CrossRef] [Medline]
    136. Badr H, Lipnick D, Diefenbach MA, Posner M, Kotz T, Miles B, et al. Development and usability testing of a web-based self-management intervention for oral cancer survivors and their family caregivers. Eur J Cancer Care (Engl) 2016 Sep;25(5):806-821 [FREE Full text] [CrossRef] [Medline]
    137. Buman MP, Epstein DR, Gutierrez M, Herb C, Hollingshead K, Huberty JL, et al. BeWell24: development and process evaluation of a smartphone. Transl Behav Med 2016 Sep;6(3):438-448 [FREE Full text] [CrossRef] [Medline]
    138. Hallberg I, Taft C, Ranerup A, Bengtsson U, Hoffmann M, Höfer S, et al. Phases in development of an interactive mobile phone-based system to support self-management of hypertension. Integr Blood Press Control 2014;7:19-28 [FREE Full text] [CrossRef] [Medline]
    139. Hong Y, Dahlke DV, Ory M, Hochhalter A, Reynolds J, Purcell NP, et al. Designing iCanFit: a mobile-enabled web application to promote physical activity for older cancer survivors. JMIR Res Protoc 2013;2(1):e12 [FREE Full text] [CrossRef] [Medline]
    140. Jongstra S, Beishuizen C, Andrieu S, Barbera M, van Dorp M, van de Groep B, et al. Development and validation of an interactive internet platform for older people: the Healthy Ageing Through Internet Counselling in the Elderly study. Telemed J E Health 2017 Dec;23(2):96-104. [CrossRef] [Medline]
    141. Shorten A, Fagerlin A, Illuzzi J, Kennedy HP, Lakehomer H, Pettker CM, et al. Developing an internet-based decision aid for women choosing between vaginal birth after cesarean and planned repeat cesarean. J Midwifery Womens Health 2015;60(4):390-400. [CrossRef] [Medline]
    142. Thompson D, Cullen KW, Boushey C, Konzelmann K. Design of a website on nutrition and physical activity for adolescents: results from formative research. J Med Internet Res 2012 Apr 26;14(2):e59 [FREE Full text] [CrossRef] [Medline]
    143. Cullen KW, Thompson D, Boushey C, Konzelmann K, Chen T. Evaluation of a web-based program promoting healthy eating and physical activity for adolescents: teen choice: food and fitness. Health Educ Res 2013 Aug;28(4):704-714 [FREE Full text] [CrossRef] [Medline]
    144. Bandura A. Health promotion by social cognitive means. Health Educ Behav 2004 Apr;31(2):143-164. [CrossRef] [Medline]
    145. Terry DJ, O'Leary JE. The theory of planned behaviour: the effects of perceived behavioural control and self-efficacy. Br J Soc Psychol 1995 Jun;34(Pt 2):199-220. [CrossRef] [Medline]
    146. Prochaska JO, Johnson S, Lee P. The transtheoretical model of behavior change. In: Shumaker SA, Riekart KA, Ockene JK, editors. The Handbook of Health Behavior Change. New York, NY: Springer Publishing Co; 1998:59-84.
    147. Fogg BJ. A behavior model for persuasive design. In: Proceedings of the 4th International Conference on Persuasive Technology. 2009 Presented at: Persuasive'09; April 26-29, 2009; Claremont, CA, USA.
    148. Abras C, Maloney-Krichmar D, Preece J. User-centered design. In: Bainbridge W, editor. Encyclopedia of Human Computer Interaction. Thousand Oaks: Sage Publications; 2004:445-456.
    149. Kidd SA, Kral MJ. Practicing participatory action research. J Couns Psychol 2005;52(2):187-195. [CrossRef]
    150. Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, et al. Framework for design and evaluation of complex interventions to improve health. Br Med J 2000 Sep 16;321(7262):694-696 [FREE Full text] [CrossRef] [Medline]
    151. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. Br Med J 2008;337:a1655 [FREE Full text] [CrossRef] [Medline]
    152. Medical Research Council. 2008. Developing and evaluating complex interventions: new guidance   URL: https://mrc.ukri.org/documents/pdf/complex-interventions-guidance/ [accessed 2019-01-28] [WebCite Cache]
    153. Holmes-Rovner M. International Patient Decision Aid Standards (IPDAS): beyond decision aids to usual design of patient education materials. Health Expect 2007 Jun;10(2):103-107 [FREE Full text] [CrossRef] [Medline]
    154. Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res 2011;13(4):e126 [FREE Full text] [CrossRef] [Medline]
    155. World Health Organization. 2018. Classification of digital health interventions v1.0: a shared language to describe the uses of digital technology for health   URL: http://apps.who.int/iris/handle/10665/260480 [accessed 2018-06-20] [WebCite Cache]
    156. Graffigna G, Barello S, Triberti S. Giving (back) a role to patients in the delivery of healthcare services: theoretical roots of patient engagement. In: Patient Engagement: A Consumer-Centered Model to Innovate Healthcare. Warsaw: De Gruyter Open; 2015:13-24.


    Abbreviations

    ACM: Association for Computing Machinery
    CeHRes: Center for eHealth Research and Disease Management
    CINAHL: Cumulative Index to Nursing and Allied Health Literature
    eHealth: electronic health
    HSP: health service providers
    HSU: health service users
    IEEE: Institute of Electrical and Electronics Engineers
    MeSH: Medical Subject Headings
    MMAT: Mixed Methods Appraisal Tool
    MRC: Medical Research Council
    OHP: Optimal Health Program
    PAR: participatory action research
    UCD: User-Centered Design


    Edited by M Benham-Hutchins; submitted 04.07.18; peer-reviewed by L van Velsen, J Apolinário-Hagen; comments to author 09.09.18; revised version received 29.10.18; accepted 10.12.18; published 22.02.19

    ©Gaye Moore, Helen Wilding, Kathleen Gray, David Castle. Originally published in Journal of Participatory Medicine (http://jopm.jmir.org), 22.02.2019.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in Journal of Participatory Medicine, is properly cited. The complete bibliographic information, a link to the original publication on http://jopm.jmir.org, as well as this copyright and license information must be included.