Published on in Vol 11, No 1 (2019): Jan-Mar

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/11474, first published .
Participatory Methods to Engage Health Service Users in the Development of Electronic Health Resources: Systematic Review

Participatory Methods to Engage Health Service Users in the Development of Electronic Health Resources: Systematic Review

Participatory Methods to Engage Health Service Users in the Development of Electronic Health Resources: Systematic Review

Review

1Mental Health Executive Services, St Vincent's Hospital, Melbourne, Fitzroy, Australia

2Department of Nursing, Faculty of Medicine, Dentistry and Health Sciences, University of Melbourne, Melbourne, Australia

3Library Service, St Vincent's Hospital Melbourne, Fitzroy, Australia

4Health and Biomedical Informatics Centre, University of Melbourne, Melbourne, Australia

5Department of Psychiatry, Faculty of Medicine, Dentistry and Health Sciences, University of Melbourne, Melbourne, Australia

*these authors contributed equally

Corresponding Author:

Gaye Moore, BN (Hons), MPH, PhD

Mental Health Executive Services

St Vincent's Hospital, Melbourne

PO Box 2900

Fitzroy, 3065

Australia

Phone: 61 392311938

Email: gaye.moore@svha.org.au


Background: When health service providers (HSP) plan to develop electronic health (eHealth) resources for health service users (HSU), the latter’s involvement is essential. Typically, however, HSP, HSU, and technology developers engaged to produce the resources lack expertise in participatory design methodologies suited to the eHealth context. Furthermore, it can be difficult to identify an established method to use, or determine how to work stepwise through any particular process.

Objective: We sought to summarize the evidence about participatory methods and frameworks used to engage HSU in the development of eHealth resources from the beginning of the design process.

Methods: We searched for studies reporting participatory processes in initial development of eHealth resources from 2006 to 2016 in 9 bibliographic databases: MEDLINE, EMBASE, CINAHL, PsycINFO, Emcare, Cochrane Library, Web of Science, ACM Guide to Computing Literature, and IEEE Xplore. From 15,117 records initially screened on title and abstract for relevance to eHealth and early participatory design, 603 studies were assessed for eligibility on full text. The remaining 90 studies were rated by 2 reviewers using the Mixed Methods Appraisal Tool Version 2011 (Pluye et al; MMAT) and analyzed with respect to health area, purpose, technology type, and country of study. The 30 studies scoring 90% or higher on MMAT were included in a detailed qualitative synthesis.

Results: Of the 90 MMAT-rated studies, the highest reported (1) health areas were cancer and mental disorders, (2) eHealth technologies were websites and mobile apps, (3) targeted populations were youth and women, and (4) countries of study were the United States, the United Kingdom, and the Netherlands. Of the top 30 studies the highest reported participatory frameworks were User-Centered Design, Participatory Action Research Framework, and the Center for eHealth Research and Disease Management (CeHRes) Roadmap, and the highest reported model underpinning development and engagement was Social Cognitive Theory. Of the 30 studies, 4 reported on all the 5 stages of the CeHRes Roadmap.

Conclusions: The top 30 studies yielded 24 participatory frameworks. Many studies referred to using participatory design methods without reference to a framework. The application of a structured framework such as the CeHRes Roadmap and a model such as Social Cognitive Theory creates a foundation for a well-designed eHealth initiative that ensures clarity and enables replication across participatory design projects. The framework and model need to be clearly articulated and address issues that include resource availability, responsiveness to change, and the criteria for good practice. This review creates an information resource for future eHealth developers, to guide the design of their eHealth resource with a framework that can support further evaluation and development.

Trial Registration: PROSPERO CRD42017053838; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=53838

J Participat Med 2019;11(1):e11474

doi:10.2196/11474

Keywords



Rationale

Individuals are increasingly being offered access to health services via electronic health (eHealth), sometimes called digital health, that is, health-related electronic resources that connect them with health service providers (HSP) over the internet. Examples include websites, portals, social media sites, serious games, mobile apps, wearable self-monitoring devices, online learning sites, telehealth platforms, and shareable electronic health records. Patients, clients, or consumers in this review are called health service users (HSU). They may require services to support their physical health, mental health, and well-being in the broadest sense of the World Health Organization’s definition (Table 1) [1].

The involvement of HSU as full participants in eHealth innovations responds to a social movement that is over a decade old and influenced by many general trends in the digital economy and the information society [2]. Regardless of the form or purpose of eHealth resources, a common question is how HSP and HSU can optimally work together to design, build, and operationalize them; monitor their performance; and evaluate their impact [3].

Like most HSU, most HSP have little or no experience or training that equips them to collaborate effectively to develop eHealth resources, and so they are likely to turn to information technology professionals. However, technical developers or vendors who are commissioned to develop eHealth initiatives and technologies will turn back to their health sector partners for answers to who, what, where, when, why and how questions about engaging HSU in the early stages of the process. Furthermore, technical developers’ responsibilities usually end on delivery of a working product. Thereafter, HSP may or may not have clear ways of assigning responsibility for managing and governing the product’s use; in any case, HSU participation may be overlooked in these later stages in the life cycle of an eHealth resource. Participatory action research (PAR) [4] may be the launchpad for development [3], but at the end of development projects, there remains the need to operationalize and sustain the eHealth resources that have been created. The continuing quality assurance of eHealth resources within the auspicing health service also needs ongoing participation by HSU.

Apart from operational needs for HSU participation, there are ethical reasons for it. HSP have an ethical responsibility for ensuring that eHealth innovations achieve health outcomes for HSU. HSP are committed to evidence-based practice, in this as in other aspects of their work. Therefore, when they think about developing and deploying new eHealth resources, where do they find what is recognized as good practice in HSU participation? There are so many case studies that it is a near-impossible task to synthesize them all; furthermore, some talk the talk but do not walk the walk of HSU participation, some do not follow any recognized methodology, and some finish early in the life cycle of the eHealth resource.

There are numerous reviews and design guidelines that generalize about theories and methods of HSU participation in eHealth design. They emphasize the importance of the following basic principles:

  • appreciation and understanding from the outset, of the range of potential HSU characteristics, goals, needs, values, and perspectives on use [5-7]
  • attention to the needs of HSU not just as individual actors but also within their formal and informal care networks [8]
  • careful alignment of diverse concerns, attitudes, and perspectives that expert content creators, HSP, and HSU may have [9-12]
  • genuine active involvement so that HSU have an opportunity to identify practical problems and design, test, evaluate, and make decisions about technology in a range of environments [13-15]

At the same time, they note that methods of HSU participation in eHealth design need to use human and other project resources judiciously. The themes are as follows:

  • ensuring that complex planning and evaluation models are able to be translated and streamlined to develop resources that are practical, feasible, and impactful in real-life settings [16]
  • taking a systematic approach to requirements specification to avoid mismatch with the organizational context and to support summative evaluation on a feature-specific level [17]
  • applying automation to expedite routine steps to create libraries of typical users and use cases and to manage unforeseen lessons learned for efficiency [18,19]
Table 1. Glossary of terms.
TermDefinition
HealthA state of complete physical, mental, and social well-being and not merely the absence of disease or infirmity, in accordance with the World Health Organization’s definition [1]
Health servicesOrganizations delivering support in the areas of physical, mental, or social well-being
Health service usersPatients, clients, or consumers who are recipients of health services. Excludes family or informal caregivers who receive health services solely in their caring role
Health service providersIncludes health professionals and health service managers
Electronic health (eHealth)Health-related electronic resources, sometimes described as innovations, initiatives, applications, solutions, and tools
eHealth resourcesInteractive electronic resources including websites, portals, mobile technology, mobile apps, blogs, social media, podcasts, wearable fitness or tracking devices, e-learning, telehealth, video, electronic health records, and software
Participatory methodsSystematic methods used to include end users as codevelopers. Includes participatory approaches, processes, and frameworks, such as participatory action research
Participatory processesSpecific steps taken to ensure engagement of end users, such as focus groups

Nevertheless, key considerations aside, it is difficult for HSP to identify from the literature a recognized, reliable methodological framework for engaging with HSU in the development of eHealth resources. A recent systematic review found that the literature variously encompassed 6 key phases and 17 different methods of participatory design, and it also found that sufficiency of reporting was poor and that no study undertook a robust assessment of efficacy [20]. This makes it difficult for HSP to study the effects of HSU participation in eHealth resources development on reach, adoption, acceptance, and efficacy of the intervention. Relative to other areas of health research, this type of study is immature, without widely endorsed methodological conventions for describing realistic aims for such projects or for determining valid measures of such effects [21].

Therefore, this paper investigates reports of eHealth applications and tools and resource development to determine what methods have been used systematically to ensure full HSU participation. We sought to distill evidence of positive, negative, or other unanticipated effects that have arisen at any stage in the eHealth resource life cycle from various HSU participation methods. Within these participatory approaches, we identified the reported impact from the point of view of HSU and HSP.

The impetus for this study began when the authors sought a strong research framework within which to undertake co-design of an eHealth initiative. The project was based on a print-based and workshop-based psychoeducational intervention called the Optimal Health Program (OHP). The authors wanted to ensure that they chose a rigorous methodological framework for redevelopment of OHP as an eHealth resource. Utilizing proven participatory methods would (1) optimize HSU engagement with the OHP resource that was developed, (2) strengthen the relevance of the resource to intended HSU, and (3) provide a logical foundation for long-term evaluation and improvement of the resource.

Objectives

This paper reviews published research reports that include detailed descriptions of participatory methods to engage HSU in eHealth resource development projects. Through synthesizing answers to the following questions, the objective of this paper is to support critical evaluation of this type of methodology and informed selection of appropriate approaches in future research and development projects:

  1. What types of eHealth resources have been developed using participatory processes, intended for what types of end users?
  2. What frameworks have been used from the very beginning of the design process to ensure participation by the intended end users in the development of eHealth resources?
  3. What methods within these frameworks have been most effective in supporting full involvement by intended end users of eHealth resources?
  4. What aspects of the participatory methods in these eHealth projects have emerged as being most important to end users?
  5. What positive, negative, or other unanticipated effects of participatory methods have the researchers reported at eHealth resource design, development, implementation, or evaluation stages?

Protocol and Registration

This systematic review has been carried out in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [22,23]. Protocol CRD42017053838 was lodged with the PROSPERO international prospective register of systematic reviews.

Information Sources

A total of 9 bibliographic databases were searched, including 6 health and biomedical databases and 3 technology databases:

  • Ovid MEDLINE(R) Epub Ahead of Print, In-Process & Other Non-Indexed Citations, Ovid MEDLINE(R) Daily and Ovid MEDLINE(R) 1946 to Present (“MEDLINE”)
  • EMBASE (Embase.com ) (“EMBASE”)
  • CINAHL Plus with Full Text (EBSCOhost) (“CINAHL”)
  • PsycINFO 1806 to January Week 4 2017 (Ovid) (“PsycINFO”)
  • Ovid Emcare 1995 to 2016 week 49 (“Emcare”)
  • Cochrane Library, including Cochrane Database of Systematic Reviews; Database of Abstracts of Reviews of Effect; Cochrane Central Register of Controlled Trials; Cochrane Methodology Register; Health Technology Assessment Database; NHS Economic Evaluation Database; About the Cochrane Collaboration (“Cochrane”)
  • Web of Science Core Collection (“Web of Science”)
  • ACM Guide to Computing Literature (“ACM”)
  • IEEE Xplore Digital Library (“IEEE”)

Additional articles were identified from reference lists of key articles and cited by references in Google Scholar.

Search

Search strategies were developed by an experienced medical research librarian (HW) in consultation with the OHP project leader (GM) and an expert eHealth researcher (KG).

In December 2015, scoping searches were developed and run in MEDLINE, EMBASE, CINAHL, PsycINFO, and Cochrane. In April 2016, brief confirmatory searches were run in Google Scholar to consider gaps in the initial strategy and additional search terms or databases that could be included. As a result, search strategies were refined and rerun in the initial health and biomedical databases as well as 3 additional technology databases: Web of Science, ACM, and IEEE. In February 2017, searches were updated to include results to the end of 2016. At this stage, an additional health database, Emcare, was also searched.

Within the health and biomedical databases (MEDLINE, EMBASE, CINAHL, PsycINFO, Emcare, and Cochrane) the search strategies combined the general concepts of user participation AND electronic resources AND program design. These search strategies were not limited to health-related conditions or resources because they yielded a small proportion of nonhealth-related results that could be removed manually. This enabled a very wide range of health conditions, HSU, organizations, and resources to be included in the results.

Within the broader technology databases that are not health specific (Web of Science, ACM, and IEEE), the search strategies were necessarily limited to health-related resources, combining the general concepts of user participation AND electronic resources AND (health OR well-being) AND program design.

A detailed search strategy was developed for MEDLINE using a combination of Medical Subject Headings (MeSH) and text words (Textbox 1). This was then adapted for the other databases, taking into account relevant subject headings and syntax. Search results were limited to publications dated from January 2006 to December 2016 and publications in English language. All database searches were updated in February 2017. Final search strategies for all databases are provided in Multimedia Appendix 1.

Search strategy for Ovid MEDLINE.

Ovid MEDLINE(R) Epub Ahead of Print, In-Process & Other Non-Indexed Citations, Ovid MEDLINE(R) Daily and Ovid MEDLINE(R) 1946 to Present

  1. Community-Based Participatory Research/ or consumer participation/ or patient participation/
  2. (codesign* or co-design* or coproduc* or co-produc* or cocreat* or co-creat* or participatory or e-collaboration or usability or focus group*).ti,ab.
  3. ((user* or patient* or consumer* or family or families or carer* or caregiver* or participant* or client* or stakeholder* or peer*) adj2 (centre* or center* or centric or involv* or participat* or partner* or activat* or experience or advisor* or includ* or inclusion or engag* or collaborat* or consult* or empower* or input* or led or focus*)).ti,ab.
  4. 1 or 2 or 3
  5. internet/ or blogging/ or social media/ or audiovisual aids/ or multimedia/ or cell phones/ or text messaging/ or webcasts as topic/ or Telemedicine/ or videoconferencing/ or educational technology/ or audiovisual aids/ or motion pictures as topic/ or multimedia/ or exp optical storage devices/ or radio/ or exp tape recording/ or exp television/ or Mobile Applications/ or Software Design/
  6. (internet or web* or online or www* or audiovisual* or audio-visual* or multimedia or multi-media or ehealth or e-health or mobile tech* or mobile phone* or mobile device* or mobile health or mhealth or m-health or cell phone or cellular phone or smartphone or app or apps or blog* or social media or social network* or facebook or podcast* or tracking device* or electronic health device* or fitbit or elearning or e-learning or wearable device* or smartwatch* or wearable electronics or telemedicine or tele-medicine or telehealth or tele-health or video* or electronic patient record* or electronic medical record* or electronic health record* or electronic record* or wiki* or portal* or behavioural intervention technolog* or health information technolog* or software or medical informatic* or health informatic* or digital health).ti.
  7. 5 or 6
  8. Program development/ or planning techniques/ or equipment Design/ or software design/
  9. (develop* or creat* or plan* or build* or implement* or codesign* or co-design*).ti,ab. or design*.ti.
  10. 8 or 9
  11. 4 and 7 and 10
  12. limit 11 to (english language and yr=“2006 -2016”)
Textbox 1. Search strategy for Ovid MEDLINE.

Study Selection

The search results were exported from all bibliographic databases to Endnote bibliographic management software. Duplicates were identified and manually removed within Endnote by HW. The records were initially screened within Endnote on title and abstract by HW, excluding results that were clearly irrelevant, that is, not electronic technology, not health or well-being related, not development processes, or not involving end users. All potentially eligible records were exported from Endnote to Covidence, an online platform for managing the systematic review process. Covidence was used to screen records on title and abstract by any 2 of GM, HW, KG, and 1 additional reviewer using predefined inclusion and exclusion criteria as shown in Textboxes 2 and 3. All types of study design were eligible for initial inclusion.

Full-text articles were obtained and uploaded to Covidence for all the available records that had been included based on title and abstract. When a number of articles reported on the same project, they were grouped into a single study to be reviewed together. The full text was reviewed independently by 2 reviewers, GM and HW, using additional inclusion and exclusion criteria (Textboxes 4 and 5).

Inclusion criteria for screening on title and abstract.
  • English language
  • Author identified
  • Abstract available
  • Intended end users are health service users (HSU)
  • Intended HSU are involved by proxies: patient associations/advocates/family caregivers
  • End users over 14 years of age
  • End users are involved in the initial design/development stages
Textbox 2. Inclusion criteria for screening on title and abstract.
Exclusion criteria for screening on title and abstract.
  • Language other than English
  • Author not identified
  • Abstract unavailable
  • Intended end users are health service providers (HSP) only
  • Intended end users are patient advocates, informal caregivers, or family caregivers in their own right
  • End users under 14 years of age
  • End users are involved only in the later stages of development
  • Dissertation
  • Duplicate record
Textbox 3. Exclusion criteria for screening on title and abstract.
Additional inclusion criteria for full-text review.
  • Full text available
  • Full conference papers
  • Sufficient information on early design/development
  • Inclusion of end users on their own behalf
  • Development of a specific electronic health (eHealth) resource
  • The eHealth resource is designed to support HSU interaction
Textbox 4. Additional inclusion criteria for full-text review.
Additional exclusion criteria for full-text review.
  • Full text unavailable
  • Conference abstract only
  • Insufficient information on early design stage
  • Patient associations/advocates/family caregivers as spokespersons for health service users (HSU)
  • No specific electronic health (eHealth) resource actually developed
  • Text or video content resources only, with no additional interactivity beyond content consumption
  • Hardware only
  • Research methodology inappropriate
  • Project aim unclear
  • Review paper only
Textbox 5. Additional exclusion criteria for full-text review.

Data Collection Process

The included studies were found to use qualitative, quantitative, and mixed methods for HSU participation in eHealth resource development; therefore, the Mixed Methods Appraisal Tool (MMAT) Version 2011 [24] was selected to analyze the rigor of these studies. The MMAT’s 19-assessment criteria were added to the extraction stage of Covidence. Each of the included full-text studies was assessed for methodological quality and rated according to the relevant MMAT criteria.

The detail of MMAT ratings under each criterion was extracted and recorded in an Excel spreadsheet. MMAT scoring metrics were used to calculate a total score for each study in order to develop a hierarchy of evidence for the strength of different methodologies (Multimedia Appendix 2).

The 2 reviewers, GM and HW, worked independently using MMAT to assess the methodological quality of papers and minimize risk of bias in assessing the literature. MMAT ratings and reasoning were compared, and conflicts were resolved through discussions between them.

Risk of Bias in Individual Studies

MMAT scores are typically 100%, 75%, 50%, and 25%. They work on the principle that a mixed-methods study is only as strong as its weakest part. This means that mixed-methods studies that have more criteria to meet (4 qualitative plus 4 quantitative plus 3 mixed method, equaling 11 criteria) could potentially be marked down more easily than studies that are purely qualitative and have fewer criteria to score (4 qualitative criteria only). In order to address this potential bias among study types, a decision was made to include an additional score of 90% to rationalize the difference that occurred between 100% and 75% in mixed-methods studies (Multimedia Appendix 2).

After assessment, studies were grouped by MMAT score and sorted into alphabetical order according to the surname of the first author. Although study numbers were initially used by the reviewers for identification purposes, these have been removed so that there is no confusion about study number and ranking. All studies with the same MMAT score hold equal ranking.

Data Items

The 90 studies assessed according to MMAT are summarized descriptively in a table (Multimedia Appendix 3). First, the data items described in Table 2 were manually extracted from the full text by HW and recorded in Excel for analysis. These results were grouped, tallied, and exported into separate tables according to characteristics of the research scope, such as health area, technology, population, or country of study (Multimedia Appendices 4-7).

Additional descriptive data were extracted from the full text of a subset of included studies, namely 30 studies that scored 90% or higher on MMAT. Data were extracted by HW and GM from the full text of each study using the data items listed in Table 3. These details were grouped, sorted, tallied, and exported into tables that summarize the main methods used to engage HSU in participatory development of eHealth resources.

Methods, frameworks, and processes varied enormously among studies; therefore, a decision was made to allocate all reported methods to the 5 stages of a single framework in order to standardize comparison. The Center for eHealth Research and Disease Management (CeHRes) Roadmap [25] was chosen for this purpose because it was specific to eHealth, highly cited (approximately 400 times between 2011 and 2017), based on the review of many eHealth and development frameworks, process oriented (not just a list of methods but a focus on specific steps), and defined within 5 stages.

Models and theories, participatory frameworks and interventions were extracted from the top 30 studies, and HW subsequently searched for additional mentions of them across the full text of the 90 MMAT–rated studies within Endnote.

Risk of Bias Across Studies

To minimize journal bias, a wide range of bibliographic databases were searched, including those with either a health focus or a technology focus. The search results were limited to English language, which could have created a cultural bias in the studies, although the 90 studies included in the quantitative analysis took place across 21 countries.

Table 2. Data items extracted from 90 studies.
VariableDefinition
Health areaMain area of health or well-being that the technology addresses; for example, cardiovascular diseases, mental disorders. Defined using MeSHa terms—a controlled vocabulary of hierarchical subject headings from MEDLINE
PurposePurpose of the technology; for example, motivation, self-care, or health education. Defined using MeSH Terms—a controlled vocabulary of hierarchical subject headings from MEDLINE
Technology typeIdentified technology developed; for example mobile app or website. If more than one, all technologies were recorded
Age groupAge group targeted by the resource (not to be confused by the age group of participants in the development process). Simplified into 3 groups: youth (12-24 years), adult (25-64 years), and aged (65+ years). eHealthb projects aimed at children under 14 years were excluded; therefore, this age group was not included
Gender specificGender specific target of a resource; for example, female only or male only. Not recorded if the resource was inclusive of all genders rather than gender specific
LGBTQI+c specificLGBTQI+ specific target of a resource; for example, men who have sex with men. Not recorded if resource was inclusive rather than LGBTQI+ specific
Cultural/multiculturalResearch focusing on a particular culture or across a number of different cultures (for example, Indigenous Australians). Not recorded if culture was not reported as an issue; for example, Swedish research taking place in Sweden with Swedish-speaking participants would not be included unless it was also researched in another country with another language for crosscultural comparison
Country where studiedCountry where the research took place. If more than one, all are included

aMeSH: Medical Subject Headings.

beHealth: electronic health.

cLGBTQI+: Lesbian, Gay, Bisexual, Transgender, Queer or Questioning, and Intersex+.

Table 3. Additional data items extracted from the top 30 studies.
VariableDefinition
Specific productSpecific resource developed; for example, named mobile app or website URL
Models and theory baseDefined structures and models within the project design and delivery, such as Stages of Change
Participatory frameworksDefined frameworks involving end users in the development of resources, such as, CeHResa Roadmap
InterventionsSpecific therapeutic program or guideline, such as Acceptance and Commitment Therapy
Health service user (HSU) populationDefined end user group for a particular eHealthb project; for example, young people with diabetes
Health service provider (HSP) populationDefined group of health professionals involved in an eHealth project; for example, mental health clinicians, oncologists
Teams or groupsDefined teams or groups involved in the development of an eHealth project; for example, leadership team, research group, or advisory group
MethodsMethods or processes used during the development of an eHealth resource. Includes both participatory and nonparticipatory methods (for example, ethics application and literature search). Participants are identified for some methods (for example, Focus Group [HSU] and Interview [HSP]_
CeHRes Roadmap stageMethods sorted into different stages of a defined participatory framework known as the CeHRes Roadmap [25]. The 5 stages include: (1) Contextual Inquiry, (2) Value Specification, (3) Design, (4) Operationalization, and (5) Summative Evaluation.
Themes/findings (HSU’s perspective)Reported feedback from HSU about the eHealth resource and development process
Author/researcher recommendationsReported results, limitations, and recommendations

aCeHRes: Center for eHealth Research and Disease Management.

beHealth: electronic health.

The development of eHealth resources is a long process, sometimes taking many years, and many publications only reported a portion of the process, with only a few reporting the entire project up to final evaluation. As conference abstracts and grey literature were excluded in favor of journal articles, sections of the development process may have been reported elsewhere but not included in our evaluation. Reference lists and cited by references in Google Scholar were searched with respect to the top 90 studies to locate connected publications reporting later stages of development, but it is possible that some publications were either missed or published after our review timeframe.


Database searches retrieved 24,674 records, which were exported to Endnote. Duplicates were removed by HW, leaving 15,117 records. These records were screened for broad relevance on title and abstract by HW and 13,096 records were excluded as clearly irrelevant. The remaining 2021 records were assessed for eligibility on title and abstract using the inclusion and exclusion criteria in Textboxes 2 and 3, and 1391 records were excluded.

The 630 remaining records were combined into 603 studies, some of which involved multiple publications. All 603 studies were assessed for eligibility on full text, and 513 studies were excluded according to the criteria in Textboxes 4 and 5, leaving 90 studies for quantitative analysis. During the screening and full text review process, 12 additional records relating to the 90 studies were identified from reference lists or contact with authors, and those records were combined into the studies. See Figure 1 for the PRISMA flow diagram.

A total of 90 studies were assessed for quality according to MMAT. Results are summarized in Table 4 and detailed results are available in Multimedia Appendix 2. An MMAT score of 100% was awarded to 28 studies and 2 studies scored 90%.

Results From 90 Studies Included in Quantitative Analysis

The 8 data items described in Table 2 were extracted from each of the 90 studies (Multimedia Appendix 3).

The major health focus of each study was grouped into a hierarchy of 18 wider MeSH subject headings, summarized in Multimedia Appendix 4. The top 5 health areas were neoplasms (cancer), mental disorders, nutritional and metabolic diseases (including weight management), virus diseases (including HIV), cardiovascular diseases, and endocrine system diseases (including diabetes).

Nine types of technology were reported in the 90 studies, and these are summarized in Multimedia Appendix 5. Websites (56 studies) and mobile apps (32 studies) were the main eHealth technologies developed. Other types of technology reported were decision tools, handheld computers, kiosk applications, personal health records, serious games, wearable devices, and telemonitoring.

Studies targeting specific populations are summarized in Multimedia Appendix 6. Of the 90 studies, 22 (24%) were youth specific, and 9 (10%) focused on the aged. Of the 90 studies, 11 (12%) reported eHealth projects for women only, and 4 (4%) were for men only. Moreover, 3 studies (3%) had a Lesbian, Gay, Bisexual, Transgender, Queer or Questioning, and Intersex+ focus. Fourteen studies (16%) had either a cultural or multicultural focus, such as a bilingual app for Indigenous Australians or the development of a website in both France and Finland.

The 90 studies took place in 21 countries, summarized in Multimedia Appendix 7. The top 6 countries were United States (33 studies), United Kingdom (15 studies), Netherlands (13 studies), Canada (7 studies), Sweden (6 studies), and Australia (6 studies). Studies also took place in Austria, Belgium, Czech Republic, Greece, Denmark, Finland, France, India, Spain, Ireland, Italy, New Zealand, Norway, Republic of Korea, and Saudi Arabia.

Results From 30 Studies Included in Qualitative Synthesis

The 30 studies scoring 90% or higher on MMAT were recorded in Excel spreadsheets and reviewed in detail. Data items listed in Table 3 were extracted for each study (Multimedia Appendix 8).

The 30 studies are listed in Table 5, along with an indication of the CeHRes Roadmap stages reported. There was often a perceived overlap between stages 1 (contextual inquiry) and 2 (value specification) such as when focus groups may have covered both stages at once. Where this appeared to happen, it was reported in the spreadsheet and included in both stages in Table 5. Where the CeHRes Roadmap was particularly useful was in highlighting stages that were often not reported, such as operationalization or summative evaluation (Table 5). It is possible that some of these studies did address each stage but did not report them in journal articles that were reviewed.

A summary of the 30 highest MMAT–rated studies is represented in Table 6 with details of the product developed, technology used and targeted population. The health area and general purpose of each eHealth project, categorized using Medical Subject Headings (MeSH) is summarized in Multimedia Appendix 9.

The methods were recorded in Excel spreadsheets using the original terminology reported in each study. The details included the number of HSU or HSP involved in each process, the order of each activity as reported, and subprocesses within each method (for example, the type of design activity or workshop activity). These details are included in Multimedia Appendix 8. These detailed methods were then grouped so that they could be summarized using a consistent terminology and then compared. This summary of methods is included for each study in Table 7.

Models and theories referred to in the top 30 studies are shown in Table 8.

Tables 9-13 give an overview of the options used to satisfy each stage of the CeHRes roadmap and the popularity of these methods. Many of the methods reported may demonstrate formative evaluation processes occurring as part of an iterative process. We recommend referring to Multimedia Appendix 8 and the original references for additional information that may be able to identify the practical steps that were implemented.

Figure 1. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram.
View this figure
Table 4. Summary of scoring of 90 studies according to the Mixed Methods Appraisal Tool Version 2011.
MMATa score (%)Studies (n=90), n (%)Articles (n=117), n (%)Qualitative studies only (n=33), n (%)Mixed-methods studies (n=57), n (%)References
10028 (31)41 (35)16 (48)12 (21)[26-66]
902 (2)3 (3)0 (0)2 (4)[67-69]
7541 (46)53 (45)11 (33)30 (53)[70-123]
5011 (12)11 (9)3 (9)8 (14)[124-134]
258 (9)9 (77)3 (9)5 (9)[5,135-143]

aMMAT: Mixed Methods Appraisal Tool Version 2011.

Lastly, the models and theories, participatory frameworks and interventions identified in the top 30 studies were searched across the full text of the 90 MMAT–rated studies within Endnote, and the results were ranked in order of prevalence in Multimedia Appendices 10-12.

Twenty three models or theories were identified from the 30 studies scoring 90% or higher on MMAT as playing a role in the development of eHealth resources. The most often reported models and theories were Social Cognitive Theory (n=4, 13%) [144], Theory of Planned Behavior (n=3, 10%) [145], Transtheoretical Model (Prochaska Stages of Change) (n=3, 10%) [146], and the Persuasive Technology Theory/Behavior Model for Persuasive Design (n=3, 10%) [147]. A large variety of other models and theories were referred to, with little overlap between studies. Full results are recorded in Multimedia Appendix 10.

A total of 24 named participatory frameworks or approaches were identified from the 30 studies scoring 90% or higher on MMAT (Multimedia Appendix 11). Only 20 of the 30 top scoring studies referred to a specific framework, with many studies referring more broadly to using participatory design or iterative design methods without reference to a particular named framework. The most often reported participatory frameworks or approaches were User-Centered Design ([UCD], n=5, 17%) [148], PAR framework (n=4, 13%) [149], CeHRes Roadmap (n=3, 10%) [25], Medical Research Council (MRC) Guide to Developing and Evaluating Complex Interventions (n=2, 7%) [150-152], and International Patient Decision Aid Standards Collaboration (n=2, 7%) [153].

Some studies referred to specific interventions that were integral to the function of the eHealth resource that was developed. Key interventions identified in Multimedia Appendix 12 were Cognitive Behavior Therapy (CBT), Behavior Change Techniques, and Mindfulness.

Table 5. Stages of the CeHRes Roadmap addressed in top 30 studies.
Study name and referencesMMATa score (%)Contextual inquiryValue specificationDesignOperationalizationSummative evaluation
Ahtinen, 2013 [26]100b
Antypas,2014 [27]100
Bengtsson, 2014 [28,29]100
Buccieri, 2015 [30]100
Clayman, 2008 [31]100
Cordova, 2015 [32]100
Dabbs, 2009 [33]100
Das, 2013 [34]100
Davies, 2015 [35,36]100
Fennell, 2016 [39,44]100
Fonda, 2010 [40,41]100
Goldenberg, 2015 [42,43]100
Heckman, 2015 [45]100
Kelders, 2013 [46]100
Lubberding, 2016 [37,38,47]100
Meyer, 2007 [48]100
Miller, 2015 [49]100
Morrison, 2015 [50]100
O\'Brien, 2016 [51]100
Peute, 2015 [52]100
Revenas, 2015 [53-55]100
Sandlund, 2015 [56]100
Schnall, 2016 [57]100
Skjoth, 2015 [58]100
Stinson, 2014 [59]100
van Bruinessen, 2014 [60,61]100
Widman, 2016 [62]100
Winterling, 2016 [63-66]100
Ennis, 2014 [67,69]90
Fleisher, 2014 [68]90

aMMAT: Mixed Methods Appraisal Tool Version 2011.

bNo information.

Table 6. Product, technology, and population in top 30 studies.
Study and referencesProductTechnologyPopulation
Ahtinen 2013 [26]Living application, a wellness app to support physical activityMobile appAdult, Finland and India
Antypas 2014 [27]Skibotn Rehabilitation Center resource with personal profile, activity calendar, and SMSb remindersWebsite and mobile appAdult, aged, Norway
Bengtsson 2014 [28,29]Mobile phone self-report system for self-management of hypertensionMobile appAdult, aged, Sweden
Buccieri 2015 [30]Supporting and Assisting Youth (SAY) mobile app for homeless youthMobile appYouth, Canada
Clayman 2008 [31]Cancercarelinks.org—Cancer Care Links for women with breast cancerWebsiteAdult, aged, female only, USAc
Cordova 2015 [32]Mobile app version of Storytelling for Empowerment (S4E)—HIV/sexually transmitted infections and drug abuse preventive intervention for primary careMobile appYouth, USA
Dabbs 2009 [33]Pocket Personal Assistant for Tracking Health (Pocket PATH) for lung transplant patientsHandheld computerYouth, adult, aged, USA
Das 2013 [34]electronic health portal for weight loss patients undergoing treatmentWebsiteYouth, adult, Norway
Davies 2015 [35,36]Hep B Story —culturally appropriate bilingual mobile app for Indigenous Australians with hepatitis BMobile appAdult, indigenous Australians, Australia
Fennell 2016 [39,44]Country Cancer Support websiteWebsiteYouth, adult, aged, Australia
Fonda 2010 [40,41]My Diabetes Data Tracker gadget—prototype personal health app for diabetes self-managementMobile app, personal health recordAdult, aged, USA
Goldenberg 2015 [42,43]HIV prevention app for men who have sex with men (MSMd)Mobile appYouth, adult, male only, MSM, USA
Heckman 2015 [45]Online skin cancer risk reduction intervention for young adults—UV4.meWebsiteYouth, USA
Kelders 2013 [46]Web-based intervention for prevention of depression, based on self-help book Living to the fullWebsiteAdult, Netherlands
Lubberding 2016 [37,38,47]OncoKompas—online self-management application for cancer survivors Oncokompas.nlWebsiteAdult, aged, Netherlands
Meyer 2007 [48]studentdepression.org—student focused website for depression self helpWebsiteYouth, UKe
Miller 2015 [49]Prostate Cancer Online Guide and Resource for Electronic Survivorship (PROGRESS)—Web-based education program for prostate cancer survivors finishing active treatmentWebsiteAdult, aged, male only, USA
Morrison 2015 [50]Living well with Asthma-online resource for self- management of asthmaWebsiteYouth, adult, UK
O’Brien 2016 [51]LEAP (Living, Eating, Activity, and Planning in retirement)—Web-based lifestyle intervention in retirementWebsiteAdult, aged, UK
Peute 2015 [52]Website for childhood cancer survivorsWebsiteYouth, Netherlands
Revenas 2015 [53-55]tRAppen —Swedish app for self-management of physical activity in rheumatoid arthritisMobile appYouth, adult, aged, Sweden
Sandlund 2015 [56]Mobile exercise app to prevent falls in senior citizensMobile appAged, Sweden
Schnall 2014 [57]HIV prevention for high risk men who have sex with men (MSM)Mobile appYouth, adult, male only, MSM, USA
Skjoth 2015 [58]Web-based decision aid support for pregnant women to make informed choices about Downs Syndrome screening-graviditetsportalen.dkWebsite, Decision ToolYouth, adult, female only, Denmark
Stinson 2014 [59]iCanCope with Pain—mobile based self- management program for youth with chronic painMobile appYouth, Canada
Van Bruinessen 2014 [60,61]PatientTIME; 3 products: self-directed online communication tool, corresponding evaluation plan, and implementation plan. Empowering patients to communicate with HSPWebsiteYouth, adult, Netherlands
Widman 2016 [62]ProjectHeartforGirls.com—interactive Web program to improve sexual communication and reduce HIV/sexually transmitted disease risk in adolescent girlsWebsiteYouth, female only, USA
Winterling 2016 [63-66]Fex-Can, fertility and sexuality following cancerWebsiteYouth, Sweden
Ennis 2014 [67,69]myhealthlocker - electronic personal health record for people with severe mental illnessPersonal Health RecordYouth, adult, UK
Fleisher 2014 [68]Web-based decision-making intervention in cancer clinical trials PRE-ACT (Preparatory Education About Clinical Trials)WebsiteAdult, aged, USA

aMeSH: Medical Subject Headings.

bSMS: short message service text messaging.

cUSA: United States of America.

dMSM: men who have sex with men.

eUK: United Kingdom.

Table 7. Participatory frameworks and summary of methods in top 30 studies.
StudyParticipatory frameworkSummary of methods
Ahtinen 2013 [26]Constructive design research, persuasive system design modelInterview (HSUa), thematic analysis, observation, think aloud, design activity, focus group (HSU), focus group (HSPb), co-design workshop (HSU + HSP), affinity wall, magical gadgets, content creation (HSU), iterative development, prototype, prototype evaluation
Antypas 2014 [27]cFocus group (HSU), thematic analysis, prototype, randomized controlled trial (RCTd)
Bengtsson 2014 [28,29]Ethics approval, HSP/expert identification of need, Focus group (HSU), focus group (HSP), thematic analysis, literature search, initial mock-up, interview (HSP), iterative development process
Buccieri 2015 [30]Spiral technology action research model, youth centered participatory action studyAdvisory group (HSU), literature search, focus group (HSU), design workshop (HSU), design activity-sketching idea, content creation (HSU), prototype, prototype evaluation, promotional campaign
Clayman 2008 [31]HSP/expert identification of needs, interview (HSU)—longitudinal series, thematic analysis, initial mock-up, iterative development process, prototype, prototype evaluation, questionnaire (HSU)
Cordova 2015 [32]Agile software development, community-based participatory research, ecodevelopmental frameworkAdvisory group, focus group (HSU), interview (HSU), thematic analysis, initial mock-up, iterative development process, prototype
Dabbs 2009 [33]User Centered DesignLiterature search, survey (HSU), interview (HSU), observation, design workshop (HSU), iterative development process, prototype, prototype evaluation, screen capture, usability testing, testing final version, usability questionnaire, RCT
Das 2013 [34]Human centered design, iterative participatory design, collaborative analysis of requirements and designEthics approval, HSP/expert identification of needs, interview (HSP), observation, thematic analysis, design workshop (HSU), design workshop (HSP), workshop evaluation, design activity—sketching ideas, interview (HSU), co-design workshop (HSU + HSP), iterative development process, prototype, prototype evaluation, think aloud, screen capture, usability testing, questionnaire (HSU), usability questionnaire, implementation at location used for case study
Davies 2015 [35,36]Participatory action research frameworkEthics approval, interview (HSU), thematic analysis, focus group (HSU), initial mock-up (storyboard), iterative development process, prototype, translation back and forth, launch event, evaluation questionnaire
Fennell 2016 [39,44]Participatory action research frameworkEthics approval, literature search, review other resources, survey (HSU), interview (HSU), thematic analysis, advisory group (HSU), iterative development process, prototype, prototype evaluation, questionnaire (HSU), promotional campaign, launch event, usage statistics collected (google analytics), feedback form /Web survey /follow up survey
Fonda 2010 [40,41]User Centered DesignFocus group (HSU), thematic analysis, focus group (HSP), iterative development process, prototype, prototype evaluation, prototype demonstration
Goldenberg 2015 [42,43]Ethics approval, focus group (HSU), focus group (HSP), interview (HSP), thematic analysis, prototype, prototype evaluation
Heckman 2015 [45]Survey (HSU), interview (HSU), think aloud, focus group (HSU), focus group (HSP), iterative development process, content development (HSP), prototype, prototype evaluation, readability/health literacy evaluation by experts, cognitive interviewing (HSU), acceptability testing, usability testing, questionnaire (HSU), thematic analysis, pilot testing, RCT
Kelders 2013 [46]CeHRes Roadmap, Human Centered DesignLiterature search, focus group (HSP), interview (HSU), thematic analysis, rapid prototyping, initial mock up, iterative development process, prototype, think aloud, usability testing, cognitive walkthrough
Lubberding 2016 [37,38,47]Ethics approval, interviews (HSU), interviews (HSP), thematic analysis, content creation (HSU), iterative development process, prototype, prototype demonstration, think aloud, usability testing, cognitive walkthrough, implementation plan, feasibility study
Meyer 2007 [48]Action Research Framework, Action Research SpiralEthics approval, advisory group, interview (HSU), survey (HSU), focus group (HSP), content creation (HSU), initial mock up, prototype, questionnaire (HSU), launch event, usage statistics collected, feedback form/Web survey/follow up survey
Miller 2015 [49]Iterative DesignLiterature search, review other resources, HSP/expert identification of needs, focus group (HSU), interview (HSU), initial mock up, content creation (HSU), iterative development process, prototype, observation, think aloud, readability/health literacy evaluation by experts, usability testing, thematic analysis, RCT
Morrison 2015 [50]User Centered Design, Medical Research Council Guide to Developing and Evaluating Complex InterventionsEthics approval, literature search, HSP/expert identification of needs, focus group (HSU), focus group (HSP), thematic analysis, initial mock-up, interview (HSU), iterative development process, prototype, think aloud, RCT
O’Brien 2016 [51]Iterative DesignEthics approval, literature search, HSP/expert identification of needs, co-design workshop (HSU + HSP), design activity, thematic analysis, content creation (HSU), iterative development process, prototype evaluation, cognitive walkthrough
Peute 2015 [52]User Centered Design, Iterative Development Methodology, Website Development Model for the Healthcare ConsumerLiterature search, survey (HSU), thematic analysis, co-design workshop (HSU + HSP), initial mockup, iterative development process, prototype, think aloud, screen capture, usability questionnaire, heuristic evaluation (experts)
Revenas 2015 [53-55]Participatory Action Research Framework, Experience Based Design, User Centered DesignEthics approval, survey (HSU), focus group (HSU), thematic analysis, advisory group (HSU), focus group (HSP), co-design workshop (HSU + HSP), iterative development process, prototype, observation
Sandlund 2015 [56]Form-IT, Participatory and Appreciative Action and Reflection, Soft Systems ThinkingEthics approval, focus group (HSU), interview (HSU), design workshop (HSU), design activity, iterative development process, prototype, prototype evaluation, observation, questionnaire (HSU), thematic analysis
Schnall 2014 [57]Information Systems Research Framework, Iterative Design, User Centered DesignLiterature search, focus group (HSU), thematic analysis, design workshop (HSU), design workshop (HSP), design activity—sketching ideas, iterative development process, prototype, usability testing, questionnaire (HSU), usability questionnaire, heuristic evaluation (experts)
Skjoth 2015 [58]CeHRes Roadmap, International Patient Decision Aid Standards CollaborationLiterature search, focus group (HSU), focus group (HSP), interview (HSU), interview (HSP), observation, thematic analysis, prototype, prototype evaluation, design reflects clinical pathway
Stinson 2014 [59]User Centered Design, Iterative designEthics approval, survey (HSU), survey (HSP), thematic analysis, focus group (HSU), focus group (HSP), interview (HSU), prototype, RCT
Van Bruinessen 2014 [60,61]Intervention Mapping Framework, User Centered Design, Context Mapping Framework, Stanford Guidelines for Web CredibilityEthics approval, advisory group (HSU), survey (HSU), focus group (HSU), design activity, thematic analysis, iterative development process, think aloud, heuristic evaluation (experts), implementation plan, RCT
Widman 2016 [62]Interview (HSU), thematic analysis, literature search, review of other resources, advisory group (HSU), focus group (HSU), content creation (HSU), iterative development process, prototype, think aloud, usability testing, RCT
Winterling 2016 [63-66]CeHRes RoadmapEthics approval, advisory group, focus group (HSU), interview (HSU), thematic analysis, iterative development process, prototype, RCT
Ennis 2014 [67,69]Ethics approval, advisory group (HSU), survey (HSU), focus group (HSU), interview (HSU), thematic analysis, prototype, prototype evaluation, feasibility study, HSU as cofacilitators
Fleisher 2014 [68]International Patient Decision Aid Standards CollaborationSurvey (HSU), focus group (HSU), interview (HSU), design activity, content creation (HSU), prototype evaluation, observation, RCT, feedback form/Web survey/follow-up survey

aHSU: health service users.

bHSP: health service providers.

cNo information.

dRCT: randomized controlled trial.

Table 8. Model or theory base in top 30 studies.
Study and referencesModel/theory base
Ahtinen 2013 [26]Prochaska Stages of Change, Social Cognitive Theory
Antypas 2014 [27]Prochaska Stages of Change, Health Behavior Theory, Social Cognitive Theory, Health Action Process Approach, Regulatory Focus Theory
Bengtsson 2014 [28,29]Common Sense Model
Buccieri 2015 [30]Critical Social Theory of Youth Empowerment
Clayman 2008 [31]a
Cordova 2015 [32]Empowerment Theory
Dabbs 2009 [33]
Das 2013 [34]
Davies 2015 [35,36]Paasche-Orlow Wolf\'s Model (causal pathways)
Fennell 2016 [39,44]Prochaska Stages of Change, Health Belief Model, Social Cognitive Theory, Reasoned Action Model, Theory of Planned Behavior, Information-Motivation Behavioral Skills Model
Fonda 2010 [40,41]
Goldenberg 2015 [42,43]
Heckman 2015 [45]Integrative Model of Behavior Prediction
Kelders 2013 [46]Persuasive Technology Theory, Business modeling
Lubberding 2016 [37,38,47]
Meyer 2007 [48]Social Constructionist Philosophy
Miller 2015 [49]Behavioral Science Theory, Cognitive-Social Health Information Processing Model
Morrison 2015 [50]Normalization Process Theory
O’Brien 2016 [51]Health Action Process Approach
Peute 2015 [52]
Revenas 2015 [53-55]
Sandlund 2015 [56]Appreciative Inquiry
Schnall 2014 [57]
Skjoth 2015 [58]--
Stinson 2014 [59]Social Learning Theory, Behavioral Activation
Van Bruinessen 2014 [60,61]
Widman 2016 [62]Fuzzy-trace Theory, Reasoned Action Model
Winterling 2016 [63-66]
Ennis 2014 [67,69]
Fleisher 2014 [68]Cognitive-Social Health Information Processing Model, Ottawa Decision Support Framework

aNo model or theory base described.

Table 9. Methods used in stage 1 (contextual enquiry) of the CeHRes Roadmap—top 30 studies.
Process/methodTotal studies (n=30), n (%)References
Ethics approval15 (50)[28,29,34-39,42-44,47,48,50,51,53-56,59-61,63-67,69]
Advisory group involved (HSUa)6 (20)[30,32,48,60,61,63-67,69]
Literature search10 (30)[30,33,39,44,46,49-52,57,58]
Review other resources2 (7)[39,44,49]
HSPb or expert identified needs6 (20)[28,29,31,34,49-51]
Survey (HSU)9 (30)[33,39,44,45,52-55,59-61,67-69]
Survey (HSP)1 (3)[59]
Focus group (HSU)13 (43)[27-30,32,40-43,50,53-58,63-67,69]
Focus group (HSP)5 (17)[28,29,42,43,46,50,58]
Interview (HSU)13 (43)[26,31-33,35-39,44,45,47,48,56,62-67,69]
Interview (HSP)3 (10)[34,37,38,42,43,47]
Observation2 (7)[33,34]
Think aloud1 (3)[45]
Thematic or data analysis18 (60)[26-29,31,32,34-44,47,50,52,57,59,62-67,69]

aHSU: health service users.

bHSP: health service providers.

Table 10. Methods used in stage 2 (value specification) of the CeHRes Roadmap—top 30 studies.
Process/methodTotal studies (n=30), n (%)References
Advisory group involved (HSUa)6 (20)[30,32,48,60,61,63-67,69]
Literature search2 (7)[28,29,62]
Review other resources1 (3)[62]
Survey (HSU)1 (3)[48]
Focus group (HSU)17 (57)[27-30,32,35,36,40-43,45,49,50,53-56,58-61,63-66,68]
Focus group (HSPb)5 (17)[28,29,42,43,48,50,59]
Interview (HSU)10 (33)[26,31,32,37-39,44,46,47,49,58,62,67,69]
Interview (HSP)1 (3)[58]
Observation3 (10)[26,33,58]
Design workshop (HSU)3 (10)[34,56,57]
Design workshop (HSP)1 (3)[34]
Co-design workshop (HSU+HSP)2 (7)[51,52]
Workshop evaluation1 (3)[34]
Think aloud1 (3)[26]
Design activity5 (17)[26,34,51,56,60,61]
Thematic or data analysis17 (57)[26,27,34,37-44,46,47,50,51,53-55,57-62,67,69]
Content creation (HSU)1 (3)[48]
Rapid prototyping1 (3)[46]
Initial draft or simple mock up9 (30)[28,29,31,32,35,36,46,48-50,52]

aHSU: health service users.

bHSP: health service providers.

Table 11. Methods used in Stage 3 (design) of the CeHRes Roadmap—top 30 studies.
Process/methodTotal studies (n=30), n (%)References
Advisory group involved (HSUa)7 (23)[30,32,39,44,53-55,60-66]
Focus group or group discussion (HSU)9 (30)[26,30,40-43,45,53-55,62-66,68]
Focus group or group discussion (HSPb)3 (10)[40-43,53-55]
Interview (HSU)10 (33)[26,31,32,34,39,44,45,49,50,59,68]
Interview (HSP)2 (7)[28,29,37,38,47]
Design workshop (HSU only)3 (10)[30,33,56]
Design workshop (HSP only)1 (3)[57]
Co-design workshop (HSU + HSP)4 (13)[26,34,51,53-55]
Design activity; for example, card sorting, sketching, affinity wall7 (23)[26,30,34,51,56,57,68]
Content creation (HSU)7 (23)[26,30,37,38,47,49,51,62,68]
Iterative design process22 (73)[26,28,29,31-41,44-47,49-57,60-66]
Prototype, mockup or storyboard26 (87)[26,27,30-50,52-59,62-67,69]
Translation1 (3)[35,36]
Testing/evaluating prototype14 (47)[26,30,31,33,34,39-45,51,56,58,67-69]
Prototype demonstration2 (7)[37,38,40,41,47]
Observation5 (17)[33,49,53-56,68]
Think aloud9 (30)[34,37,38,45-47,49,50,52,60-62]
Screen capture or recording3 (10)[33,34,52]
Readability pr health literacy evaluation2 (7)[45,49]
Usability testing8 (27)[33,34,37,38,45-47,49,57,62]
Cognitive walkthrough3 (10)[37,38,46,47,51]
Survey/questionnaire (HSU)7 (23)[31,34,39,44,45,48,56,57]
Usability questionnaire3 (10)[34,52,57]
Heuristic evaluation3 (10)[52,57,60,61]
Thematic or data analysis14 (47)[26,28,29,37,38,42,43,45-47,49-57,59]

aHSU: health service users.

bHSP: health service providers.

Table 12. Methods used in stage 4 (operationalization) of CeHRes Roadmap—top 30 studies.
Process/methodTotal studies (n=30), n (%)References
Advisory group involved (HSUa)4 (13)[30,39,44,60,61,67,69]
Design reflects clinical pathway1 (3)[58]
Implementation plan2 (7)[37,38,47,60,61]
Feasibility study2 (7)[37,38,47,67,69]
Promotional campaign2 (7)[30,39,44]
HSU as cofacilitators1 (3)[67,69]
Launch event3 (10)[35,36,39,44,48]

aHSU: health service users.

Table 13. Methods used in stage 5 (summative evaluation) of CeHRes Roadmap—top 30 studies.
Process/methodTotal studies (n=30), n (%)References
Test of final version1 (3)[33]
Evaluation questionnaire1 (3)[35,36]
Usability questionnaire1 (3)[33]
Pilot testing1 (3)[45]
Randomized controlled trial10 (33)[27,33,45,49,50,59-66,68]
Usage statistics and Google Analytics2 (7)[39,44,48]
Feedback form, Web survey or follow-up survey3 (10)[39,44,48,68]

Overview

In the era of digital health, we have a plethora of literature describing the need for better engagement with HSU to improve health care and health services, and we have access to the technologies to create a broad array of websites and mobile apps, but we lack detailed protocols for designing eHealth resources. This systematic review explored the participatory methods and frameworks used to engage HSU in the development of eHealth resources throughout the design process. UCD was most commonly reported but varied in its application and intention. Participatory methods promoting HSU engagement ranged from brief consultation via a review process to genuine collaboration, which included additional responsibility for the HSU in the actual creation process. Research and development projects that describe a conceptual model (such as Social Cognitive Theory) and a structured framework (such as the CeHRes Roadmap, which includes a diagram/flowchart) lay the foundations for us to gain greater insight into how particular processes lead to efficacious and effective eHealth resources.

Electronic Health Initiatives Developed and the Characteristics of Health Service Users

There have been extensive eHealth initiatives to address the issues of accessibility, engagement, health literacy, data collection, health promotion, early intervention, motivation, and behavioral change. Of the 90 MMAT-scored studies, websites and mobile apps make up the majority of eHealth initiatives presented in this review (Multimedia Appendix 5) with a strong multicultural focus (Multimedia Appendix 6). The end users of these eHealth initiatives were young adults, women, and the elderly (Multimedia Appendix 6) with the focus on cancer and mental health (Multimedia Appendix 4). The app has become an engagement tool used by HSP to make health information and health planning more interactive, interesting, and fun for HSU [30,32,35,36,42,43,56]. Moreover, participatory design is thought to enable young people to be creative and have substantial input into the resource development [30].

Participatory Frameworks

Analyzing the procedural frameworks used in our included studies, we found that no 2 studies reported their processes in the same way. The frameworks governing consumer participation were varied with the most reported being UCD, PAR Framework, CeHRes Roadmap, and MRC Guide to Developing and Evaluating Complex Interventions (Multimedia Appendix 11). The methods implemented to seek the HSU perspectives were also varied with the most reported being focus groups, surveys, interviews, prototype/storyboards, think aloud, and literature search (Tables 9-13). Theories and models that influenced procedures most commonly included cognition, behaviors, processes of change, motivation, and empowerment (Multimedia Appendix 10).

The diversity in eHealth initiatives supports creativity, and to ensure validity and strengthen eHealth research, there is a need to integrate a set of protocols for HSU participation and reporting guidelines [154] available via the Enhancing the QUAlity and Transparency Of health Research Network. This would not constrain methodological innovation and would allow a more effective meta-analysis and comparison of participatory development studies.

Methods Used in the Development of Resources

This review looked for evidence of sound methods for engaging HSU during the development of eHealth apps, tools, and resources. We found relatively few reports that described HSU participation throughout development (ie, from contextual enquiry to summative evaluation, Table 5). Furthermore, many of these reports did not provide adequate details according to mixed-methods appraisal standards. As shown in Multimedia Appendix 2, studies out of 603 full texts reviewed met all of our inclusion criteria and scored 90% or higher according to MMAT. This suggests that research training, funding, and dissemination agencies need to attach far greater importance to reports that describe methods more rigorously.

Others have observed that “The diverse communities working in digital health—including government stakeholders, technologists, clinicians, implementers, network operators, researchers, donors—have lacked a mutually understandable language with which to assess and articulate functionality” [155]. Tables 9-13 illustrates how deeply this lack has affected the production of cohesive research evidence, that is, it is virtually impossible to map the semantic relationships among the methodology elements to inform the discourse about what forms of participatory eHealth design work and why. Many methods are generic to human computer interaction, some take a broad behavioral approach and some include methods of measuring health outcomes in the particular area of health where the intervention is directed. One possible view is that this illustrates a flourishing of innovation and creativity. Another is that this creates a minefield for research training and peer reviewing and may represent a considerable waste of research resources.

Analyzing the conceptual bases for the methods used in the 30 studies scoring 90% or higher on MMAT, we found much variety with 23 different models or theories reported (Multimedia Appendix 10). The most commonly occurring theories were Social Cognitive Theory, Theory of Planned Behavior, Transtheoretical Model, Persuasive Technology Theory, and Health Behavior Theory. This finding offers a sound basis in evidence for future researchers who wish to follow these precedents. However, we note that research in this area has not been informed by other potentially relevant theories (for example, theories that may account better for healthcare consumers’ economic, emotional, or empowerment motives for engagement) [156].

Effective Involvement of Health Service Users

This review looked for evidence about the effectiveness of particular approaches in terms of supporting involvement by HSU. Winterling reported strategies implemented to address engagement with HSU, including 1-person central contact, established expectation of roles, compensation for time, reaching a common agreement, and HSU seen as experts on patient perspective [63-66].

It is also possible to reflect on the richness of the findings generated by particular approaches. As shown in Multimedia Appendix 13, each study reported between 2 and 10 major thematic outputs. Reports with relatively concise outputs were Bengtsson [28,29] using participatory research design and O’Brien [51] describing an array of approaches. The most extensive review was reported by Fleisher [68] using the Ottawa Decision Support Framework and participatory design and Goldenberg [42,43] using 3 types of iterative qualitative research approaches. In assessing effectiveness this way, unknown factors may be in play, such as sophistication of the data collection procedures, analytical expertise of the researchers, editorial constraints on reporting results, and temporal pressures on publication.

Important Aspects of Participatory Methods for Health Service Users

There were consistent themes that represented HSU priorities in eHealth initiatives across the selected 30 studies represented in Multimedia Appendix 13. Access to relevant, simple, and clear health information was reported consistently across most of the studies highlighting the importance of this information to make informed decisions in a timely manner. A well-designed eHealth resource that includes a framework supporting HSU involvement can significantly impact health literacy for both HSU and HSP. HSU involvement with the development of an eHealth resource created a collaborative process that required transparency and respect as well as clear mediation processes [53-55].

Being involved in the development of an eHealth resource created the opportunity for HSU to clarify the user perspective and support the relevance of the final product. Despite the variety of websites and apps, HSU reported the need for improved access to information, coordination of care, interactivity with information provided, culturally specific information, patient education, and self-management. HSU also acknowledged the importance of confidentiality and privacy when exchanging personal health information over electronic networks.

Impact of Participatory Methods Reported by Researchers

The researchers reported a number of key issues highlighting the importance of participatory methods in creating an eHealth resource that was relevant to HSU. In Multimedia Appendix 13, an outline of the research recommendations was documented for the selected 30 studies. Researchers reported on the importance of utilizing a participatory design, which included an iterative process that increased the responsiveness and relevance of their eHealth initiatives. Having the HSU perspective from the beginning was important as well as ensuring that the process was genuinely collaborative with all participants respected and acknowledged. Utilizing a health behavior theory in combination with a participatory design was noted to enhance the eHealth resource. The theory base acknowledges the importance of motivation, empowerment, and stages of change in supporting the engagement and utilization of the eHealth resource. It was also noted that the eHealth resource needed to be interesting, engaging, and in some instances include a game-playing element. Creating a more positive approach enabled the HSU to engage with serious and difficult health issues and explore options for improved health. Not only did the eHealth resource need to be interesting but it also importantly needed to be intuitive and simple to navigate.

Heckman [45] reported that their eHealth initiative was guided by intervention development, assessment guidelines for behavioral therapy, and health communication programs with health literacy best-practice. Utilizing a participatory design appeared to improve the relevance of the eHealth resource by addressing issues of culture, gender, age, and sexuality (Multimedia Appendix 6). Goldenberg [42,43] reported personalization along with interactive functionality promoted ownership for HSU. A majority of projects included both HSU and HSP in participatory methods across different developmental stages from contextual inquiry to summative evaluation of the project [28,29,31,34-38,40-43,45-55,58-61,63-67,69]. Evaluation is an integral part of participatory methodology; however, this was reported inconsistently across the 30 studies (Multimedia Appendix 8). The inclusion of a standardized tool to evaluate processes and outcomes from the HSU perspective, as part of a participatory framework, may address the need to bring more objectivity to evaluating various studies.

The demand on time and financial resources to implement a participatory design was noted by some researchers [27,30,42,43,57,68]. Availability of resources was an important consideration throughout the design process, which was often iterative. With the rapid change in technology, there is an increasing demand for HSP to be agile and develop eHealth resources more quickly but still maintaining an evidence-based, best-practice approach inclusive of HSU participation.

Limitations

A limitation of our final dataset is that because of the number of papers retrieved, we decided to limit our analysis to published journal articles and to leave out full papers in conference proceedings. It is possible that there are strong participatory processes that have not been reported in detail, or at all, in the journal literature. Moreover, we did not include studies published in languages other than English and therefore we cannot be certain that our dataset reflects work being done around the world.

As our focus was on the inclusion of HSU from the early development process onward, some studies were included that did not extend to a final evaluation of the product, and it was not always possible to consider the success or otherwise of the final eHealth product. As a part of our inclusion criteria, we required some evidence that a specific eHealth product was ultimately created or likely to be taken to completion.

A limitation of our data analysis is that MMAT is a critical appraisal tool to assess the methodological quality of studies. It does not assess the quality of the writing or the content of the research; therefore, it is possible that we have overlooked papers that may be of high quality in other respects but which we have not rated highly here because of the way their methods sections are presented. For example, under MMAT, a paper will not score highly if it does not discuss the impact of the research or report the limitations of a mixed-methods study. The studies may not have rated highly under MMAT if they used both qualitative and quantitative methods but did not acknowledge that this constituted a mixed-methods study or if only selected aspects were reported. For example, a study that reported HSU participation only at the summative evaluation stage may have involved HSU earlier as well, but this would not register in our search results because we looked for descriptions of methods for HSU participation from initial design stages.

Although categorizing all reported methods in these studies according to the 5 stages of the CeHRes Roadmap [25] was a generally useful way to compare processes across studies, absolute consistency was not achievable because of the wide variety of structuring reports, the differing terminology and naming conventions used for similar methods, and the difficulty in allocating all methods accurately to a particular process stage.

Conclusions

Agility of eHealth development is problematic in comparison to nonmedical industries as we seek to ensure safety and quality of care for HSU. It is a challenge for eHealth development to follow rigorous methods within a timeframe that responds to current needs, limited resources, and rapid technological changes. Methodological approaches to developing eHealth resources vary but the importance of engaging HSU in participatory design is consistently emphasized. By synthesizing the existing evidence about strong mixed methods for participatory development of eHealth resources, we anticipate that this systematic review will provide others with clearer guidance to plan more rapid and better-structured work of this kind.

Acknowledgments

The authors wish to thank Mark Merolli, Honorary Fellow Health and Biomedical Informatics Center, The University of Melbourne for his expertise and assistance. The authors also wish to thank Belinda Muscat and Zali Annersley, students and honorary St Vincent’s Hospital members for their assistance. This study was supported in part by a University of Melbourne Engagement grant.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Search strategies for all databases.

PDF File (Adobe PDF File), 239KB

Multimedia Appendix 2

Scoring of 90 studies according to the Mixed Methods Appraisal Tool Version 2011.

PDF File (Adobe PDF File), 471KB

Multimedia Appendix 3

Descriptive summary of 90 studies.

PDF File (Adobe PDF File), 320KB

Multimedia Appendix 4

Major health focus in 90 studies.

PDF File (Adobe PDF File), 186KB

Multimedia Appendix 5

Electronic health technology developed in 90 studies.

PDF File (Adobe PDF File), 179KB

Multimedia Appendix 6

Targeted populations in 90 studies.

PDF File (Adobe PDF File), 176KB

Multimedia Appendix 7

Countries where study took place in 90 studies.

PDF File (Adobe PDF File), 183KB

Multimedia Appendix 8

Detail of data extracted from 30 studies.

PDF File (Adobe PDF File), 264KB

Multimedia Appendix 9

Health area and purpose of product in top 30 studies categorised using Medical Subject Headings (MeSH).

PDF File (Adobe PDF File), 195KB

Multimedia Appendix 10

Model or theory base in top 30 studies.

PDF File (Adobe PDF File), 202KB

Multimedia Appendix 11

Participatory frameworks and approaches in top 30 studies.

PDF File (Adobe PDF File), 201KB

Multimedia Appendix 12

Interventions in top 30 studies.

PDF File (Adobe PDF File), 184KB

Multimedia Appendix 13

Themes and recommendations in top 30 studies.

PDF File (Adobe PDF File), 435KB

References

  1. World Health Organization. Constitution of the World Health Organization   URL: http://www.who.int/governance/eb/who_constitution_en.pdf [accessed 2018-06-20] [WebCite Cache]
  2. Okun S, Caligtan C. The engaged ePatient. In: Nelson R, Staggers N, editors. Health Informatics: An Interprofessional Approach. St Louis, Missouri: Elsevier; 2017:204-219.
  3. World Health Organization. 2016. Monitoring and evaluating digital health interventions: a practical guide to conducting research and assessment   URL: http://apps.who.int/iris/bitstream/10665/252183/1/9789241511766-eng.pdf [accessed 2019-01-17] [WebCite Cache]
  4. Loewenson R, Laurell AC, Hogstedt C, D'Ambruoso L, Shroff Z. Participatory Action Research in Health Systems: A Methods Reader. Harare, Zimbabwe: EQUINET; 2014.
  5. Asan O, Tyszka J, Fletcher KE. Capturing the patients' voices: planning for patient-centered electronic health record use. Int J Med Inform 2016 Dec;95:1-7 [FREE Full text] [CrossRef] [Medline]
  6. Gausepohl KA, Winchester WW, Smith-Jackson TL, Kleiner BM, Arthur JD. A conceptual model for the role of storytelling in design: leveraging narrative inquiry in user-centered design (UCD). Health Technol 2016 Jan 6;6(2):125-136. [CrossRef]
  7. Toraldo ML, Mangia G, Consiglio S, Mercurio R. A model for value creation: key role of the users as co-producers of e-health services. In: Proceedings IFKAD 2013. 2013 Presented at: International Forum on Knowledge Asset Dynamics; June 12-14, 2013; Zagreb, Croatia p. 1057-1070.
  8. Wherton J, Sugarhood P, Procter R, Hinder S, Greenhalgh T. Co-production in practice: how people with assisted living needs can help design and evolve technologies and services. Implement Sci 2015 May 26;10:75 [FREE Full text] [CrossRef] [Medline]
  9. Anderson JA, Godwin KM, Saleem JJ, Russell S, Robinson JJ, Kimmel B. Accessibility, usability, and usefulness of a web-based clinical decision support tool to enhance provider-patient communication around self-management to prevent (STOP) stroke. Health Informatics J 2014 Dec;20(4):261-274. [CrossRef] [Medline]
  10. Banna S, Hasan H, Dawson P. Understanding the diversity of user requirements for interactive online health services. Int J Healthcare Technol Manag 2016;15(3):253-271. [CrossRef]
  11. Crossley C, Fanfarelli J, McDaniel R. User experience design considerations for healthcare games and applications. In: 2016 IEEE International Conference on Serious Games and Applications for Health (SeGAH). 2016 Presented at: IEEE International Conference on Serious Games and Applications for Health; May 11-13, 2016; Orlando, Florida, USA p. 11-13. [CrossRef]
  12. Palm E, Nordgren A, Verweij M, Collste G. Ethically sound technology? Guidelines for interactive ethical assessment of personal health monitoring. Stud Health Technol Inform 2013;187:105-114. [CrossRef] [Medline]
  13. Clemensen J, Rothmann MJ, Smith AC, Caffery LJ, Danbjorg DB. Participatory design methods in telemedicine research. J Telemed Telecare 2016 Jan 1;23(9):780-785. [CrossRef] [Medline]
  14. Kushniruk A, Nøhr C. Participatory design, user involvement and health IT evaluation. In: Ammenwerth E, Rigby M, editors. Evidence-Based Health Informatics: Promoting Safety and Efficiency through Scientific Methods and Ethical Policy. Amsterdam: IOS Press; 2016:139-151.
  15. Park J, Kim JH, Park EJ, Ham SM. Analyzing user experience design of mobile hospital applications using the evaluation grid method. Wirel Pers Commun 2016 Dec;91(4):1591-1602. [CrossRef]
  16. Smits R, Bryant J, Sanson-Fisher R, Tzelepis F, Henskens F, Paul C, et al. Tailored and integrated web-based tools for improving psychosocial outcomes of cancer patients: the DoTTI development framework. J Med Internet Res 2014 Mar 14;16(3):11-23 [FREE Full text] [CrossRef] [Medline]
  17. van Velsen L, Wentzel J, van Gemert-Pijnen JE. Designing eHealth that matters via a multidisciplinary requirements development approach. JMIR Res Protoc 2013;2(1):e21 [FREE Full text] [CrossRef] [Medline]
  18. Hagedorn TJ, Krishnamurty S, Grosse IR. An information model to support user-centered design of medical devices. J Biomed Inform 2016 Dec;62:181-194 [FREE Full text] [CrossRef] [Medline]
  19. LeRouge C, Ma J, Sneha S, Tolle K. User profiles and personas in the design and development of consumer health technologies. Int J Med Inform 2013 Nov;82(11):e251-e268. [CrossRef] [Medline]
  20. Eyles H, Jull A, Dobson R, Firestone R, Whittaker R, te Morenga L, et al. Co-design of mHealth delivered interventions: a systematic review to assess key methods and processes. Curr Nutr Rep 2016 Jul 4;5(3):160-167. [CrossRef]
  21. DeSmet A, Thompson D, Baranowski T, Palmeira A, Verloigne M, de Bourdeaudhuij I. Is participatory design associated with the effectiveness of serious digital games for healthy lifestyle promotion? A meta-analysis. J Med Internet Res 2016 Apr 29;18(4):e94 [FREE Full text] [CrossRef] [Medline]
  22. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Ann Intern Med 2009 Aug 18;151(4):W65-W94. [CrossRef] [Medline]
  23. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009 Jul 21;6(7):e1000097 [FREE Full text] [CrossRef] [Medline]
  24. Pluye P, Robert E, Cargo M, Bartlett G, O'Cathain A, Griffiths F, et al. Mixed Methods Appraisal Tool (MMAT) – Version 2011. Montreal, Canada: Department of Family Medicine, McGill University; 2011.   URL: http:/​/mixedmethodsappraisaltoolpublic.​pbworks.com/​w/​file/​fetch/​84371689/​MMAT%202011%20criteria%20and%20tutorial%202011-06-29updated2014.​08.​21.​pdf [accessed 2019-01-17] [WebCite Cache]
  25. van Gemert-Pijnen JE, Nijland N, van Limburg M, Ossebaard HC, Kelders SM, Eysenbach G, et al. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res 2011;13(4):e111 [FREE Full text] [CrossRef] [Medline]
  26. Ahtinen A, Isomursu M, Ramiah S, Blom J. Advise, acknowledge, grow and engage: design principles for a mobile wellness application to support physical activity. Int J Hum Comput Int 2013;5(4):20-55. [CrossRef]
  27. Antypas K, Wangberg SC. Combining users' needs with health behavior models in designing an internet- and mobile-based intervention for physical activity in cardiac rehabilitation. JMIR Res Protoc 2014;3(1):e4 [FREE Full text] [CrossRef] [Medline]
  28. Bengtsson U, Kasperowski D, Ring L, Kjellgren K. Developing an interactive mobile phone self-report system for self-management of hypertension. Part 1: patient and professional perspectives. Blood Press 2014 Oct;23(5):288-295 [FREE Full text] [CrossRef] [Medline]
  29. Bengtsson U, Kjellgren K, Höfer S, Taft C, Ring L. Developing an interactive mobile phone self-report system for self-management of hypertension. Part 2: content validity and usability. Blood Press 2014 Oct;23(5):296-306 [FREE Full text] [CrossRef] [Medline]
  30. Buccieri K, Molleson G. Empowering homeless youth: building capacity through the development of mobile technology. J Community Pract 2015 May 18;23(2):238-254. [CrossRef]
  31. Clayman ML, Boberg EW, Makoul G. The use of patient and provider perspectives to develop a patient-oriented website for women diagnosed with breast cancer. Patient Educ Couns 2008 Sep;72(3):429-435. [CrossRef] [Medline]
  32. Cordova D, Bauermeister JA, Fessler K, Delva J, Nelson A, Nurenberg R, et al. A community-engaged approach to developing an mHealth HIV/STI and drug abuse preventive intervention for primary care: a qualitative study. JMIR Mhealth Uhealth 2015;3(4):e106 [FREE Full text] [CrossRef] [Medline]
  33. Dabbs AD, Myers BA, McCurry KR, Dunbar-Jacob J, Hawkins RP, Begey A, et al. User-centered design and interactive health technologies for patients. Comput Inform Nurs 2009;27(3):175-183 [FREE Full text] [CrossRef] [Medline]
  34. Das A, Svanæs D. Human-centred methods in the design of an e-health solution for patients undergoing weight loss treatment. Int J Med Inform 2013 Nov;82(11):1075-1091. [CrossRef] [Medline]
  35. Davies J, Bukulatjpi S, Sharma S, Caldwell L, Johnston V, Davis JS. Development of a culturally appropriate bilingual electronic app about hepatitis B for indigenous Australians: towards shared understandings. JMIR Res Protoc 2015 Jun 10;4(2):e70 [FREE Full text] [CrossRef] [Medline]
  36. Davies J, Bukulatjpi S, Sharma S, Davis J, Johnston V. "Only your blood can tell the story"--a qualitative research study using semi-structured interviews to explore the hepatitis B related knowledge, perceptions and experiences of remote dwelling Indigenous Australians and their health care providers in northern Australia. BMC Public Health 2014 Nov 28;14:1233 [FREE Full text] [CrossRef] [Medline]
  37. Duman-Lubberding S, van Uden-Kraan CF, Jansen F, Witte BI, van der Velden LA, Lacko M, et al. Feasibility of an eHealth application "OncoKompas" to improve personalized survivorship cancer care. Support Care Cancer 2016 May;24(5):2163-2171 [FREE Full text] [CrossRef] [Medline]
  38. Duman-Lubberding S, van Uden-Kraan CF, Peek N, Cuijpers P, Leemans CR, Verdonck-de Leeuw IM. An eHealth application in head and neck cancer survivorship care: health care professionals' perspectives. J Med Internet Res 2015 Oct 21;17(10):e235 [FREE Full text] [CrossRef] [Medline]
  39. Fennell KM, Turnbull DA, Bidargaddi N, McWha JL, Davies M, Olver I. The consumer-driven development and acceptability testing of a website designed to connect rural cancer patients and their families, carers and health professionals with appropriate information and psychosocial support. Eur J Cancer Care (Engl) 2017 Sep;26(5). [CrossRef] [Medline]
  40. Fonda SJ, Kedziora RJ, Vigersky RA, Bursell S. Combining iGoogle and personal health records to create a prototype personal health application for diabetes self-management. Telemed J E Health 2010 May;16(4):480-489. [CrossRef] [Medline]
  41. Fonda SJ, Kedziora RJ, Vigersky RA, Bursell S. Evolution of a web-based, prototype personal health application for diabetes self-management. J Biomed Inform 2010 Oct;43(5 Suppl):S17-S21 [FREE Full text] [CrossRef] [Medline]
  42. Goldenberg T, McDougal SJ, Sullivan PS, Stekler JD, Stephenson R. Preferences for a mobile HIV prevention app for men who have sex with men. JMIR Mhealth Uhealth 2014;2(4):e47 [FREE Full text] [CrossRef] [Medline]
  43. Goldenberg T, McDougal SJ, Sullivan PS, Stekler JD, Stephenson R. Building a mobile HIV prevention app for men who have sex with men: an iterative and community-driven process. JMIR Public Health Surveill 2015;1(2):e18 [FREE Full text] [CrossRef] [Medline]
  44. Gunn K, Turnbull D, McWha JL, Davies M, Olver I. Psychosocial service use: a qualitative exploration from the perspective of rural Australian cancer patients. Support Care Cancer 2013 Sep;21(9):2547-2555. [CrossRef] [Medline]
  45. Heckman C, Darlow S, Munshi T, Caruso C, Ritterband L, Raivitch S, et al. Development of an internet intervention to address behaviors associated with skin cancer risk among young adults. Internet Interv 2015 Sep;2(3):340-350 [FREE Full text] [CrossRef] [Medline]
  46. Kelders SM, Pots WT, Oskam MJ, Bohlmeijer ET, van Gemert-Pijnen JE. Development of a web-based intervention for the indicated prevention of depression. BMC Med Inform Decis Mak 2013;13:26 [FREE Full text] [CrossRef] [Medline]
  47. Lubberding S, van Uden-Kraan CF, te Velde EA, Cuijpers P, Leemans CR, Verdonck-de Leeuw IM. Improving access to supportive cancer care through an eHealth application: a qualitative needs assessment among cancer survivors. J Clin Nurs 2015 May;24(9-10):1367-1379. [CrossRef] [Medline]
  48. Meyer D. Online self-help: developing a student-focused website for depression. Couns Psychother Res 2007 Sep;7(3):151-156. [CrossRef]
  49. Miller SM, Hudson SV, Hui SA, Diefenbach MA, Fleisher L, Raivitch S, et al. Development and preliminary testing of PROGRESS: a Web-based education program for prostate cancer survivors transitioning from active treatment. J Cancer Surviv 2015 Sep;9(3):541-553 [FREE Full text] [CrossRef] [Medline]
  50. Morrison D, Mair FS, Chaudhuri R, McGee-Lennon M, Thomas M, Thomson NC, et al. Details of development of the resource for adults with asthma in the RAISIN (randomized trial of an asthma internet self-management intervention) study. BMC Med Inform Decis Mak 2015 Jul 28;15:57 [FREE Full text] [CrossRef] [Medline]
  51. O'Brien N, Heaven B, Teal G, Evans EH, Cleland C, Moffatt S, et al. Integrating evidence from systematic reviews, qualitative research, and expert knowledge using co-design techniques to develop a web-based intervention for people in the retirement transition. J Med Internet Res 2016 Aug 3;18(8):e210 [FREE Full text] [CrossRef] [Medline]
  52. Peute LW, Knijnenburg SL, Kremer LC, Jaspers MW. A concise and practical framework for the development and usability evaluation of patient information websites. Appl Clin Inform 2015;6(2):383-399 [FREE Full text] [CrossRef] [Medline]
  53. Revenäs A, Martin C, Opova CH, Brusewitz M, Keller C, Åsenlöf P. A mobile internet service for self-management of physical activity in people with rheumatoid arthritis: challenges in advancing the co-design process during the requirements specification phase. JMIR Res Protoc 2015 Sep 17;4(3):e111 [FREE Full text] [CrossRef] [Medline]
  54. Revenäs A, Opava CH, Åsenlöf P. Lead users' ideas on core features to support physical activity in rheumatoid arthritis: a first step in the development of an internet service using participatory design. BMC Med Inform Decis Mak 2014 Mar 22;14:21 [FREE Full text] [CrossRef] [Medline]
  55. Revenäs A, Opava CH, Martin C, Demmelmaier I, Keller C, Åsenlöf P. Development of a web-based and mobile app to support physical activity in individuals with rheumatoid arthritis: results from the second step of a co-design process. JMIR Res Protoc 2015;4(1):e22 [FREE Full text] [CrossRef] [Medline]
  56. Sandlund M, Lindgren H, Pohl P, Melander-Wikman A, Bergvall-Kareborn B, Lundin-Olsson L. Towards a mobile exercise application to prevent falls: a participatory design process. Int J Child Health Hum Dev 2016;9(3):389-398 [FREE Full text]
  57. Schnall R, Rojas M, Bakken S, Brown W, Carballo-Dieguez A, Carry M, et al. A user-centered model for designing consumer mobile health (mHealth) applications (apps). J Biomed Inform 2016 Apr;60:243-251. [CrossRef] [Medline]
  58. Skjøth MM, Hansen HP, Draborg E, Pedersen CD, Lamont RF, Jørgensen JS. Informed choice for participation in Down syndrome screening: development and content of a web-based decision aid. JMIR Res Protoc 2015;4(3):e113 [FREE Full text] [CrossRef] [Medline]
  59. Stinson JN, Lalloo C, Harris L, Isaac L, Campbell F, Brown S, et al. iCanCope with Pain™: user-centred design of a web- and mobile-based self-management program for youth with chronic pain based on identified health care needs. Pain Res Manag 2014;19(5):257-265 [FREE Full text] [Medline]
  60. van Bruinessen IR, van Weel-Baumgarten EM, Gouw H, Zijlstra JM, Albada A, van Dulmen S. Barriers and facilitators to effective communication experienced by patients with malignant lymphoma at all stages after diagnosis. Psychooncology 2013 Dec;22(12):2807-2814. [CrossRef] [Medline]
  61. van Bruinessen IR, van Weel-Baumgarten EM, Snippe HW, Gouw H, Zijlstra JM, van Dulmen S. Active patient participation in the development of an online intervention. JMIR Res Protoc 2014;3(4):e59 [FREE Full text] [CrossRef] [Medline]
  62. Widman L, Golin CE, Noar SM, Massey J, Prinstein MJ. ProjectHeartforGirls.com: development of a web-based HIV/STD prevention program for adolescent girls emphasizing sexual communication skills. AIDS Educ Prev 2016 Oct;28(5):365-377 [FREE Full text] [CrossRef] [Medline]
  63. Winterling J, Wiklander M, Obol CM, Lampic C, Eriksson LE, Pelters B, et al. Development of a self-help web-based intervention targeting young cancer patients with sexual problems and fertility distress in collaboration with patient research partners. JMIR Res Protoc 2016 Apr 12;5(2):e60 [FREE Full text] [CrossRef] [Medline]
  64. Armuand GM, Wettergren L, Rodriguez-Wallberg KA, Lampic C. Women more vulnerable than men when facing risk for treatment-induced infertility: a qualitative study of young adults newly diagnosed with cancer. Acta Oncol 2015 Feb;54(2):243-252. [CrossRef] [Medline]
  65. Jervaeus A, Nilsson J, Eriksson LE, Lampic C, Widmark C, Wettergren L. Exploring childhood cancer survivors' views about sex and sexual experiences-findings from online focus group discussions. Eur J Oncol Nurs 2016 Feb;20:165-172 [FREE Full text] [CrossRef] [Medline]
  66. Nilsson J, Jervaeus A, Lampic C, Eriksson LE, Widmark C, Armuand GM, et al. 'Will I be able to have a baby?' Results from online focus group discussions with childhood cancer survivors in Sweden. Hum Reprod 2014 Dec;29(12):2704-2711 [FREE Full text] [CrossRef] [Medline]
  67. Ennis L, Robotham D, Denis M, Pandit N, Newton D, Rose D, et al. Collaborative development of an electronic personal health record for people with severe and enduring mental health problems. BMC Psychiatry 2014;14(1):305 [FREE Full text] [CrossRef] [Medline]
  68. Fleisher L, Ruggieri DG, Miller SM, Manne S, Albrecht T, Buzaglo J, et al. Application of best practice approaches for designing decision support tools: the preparatory education about clinical trials (PRE-ACT) study. Patient Educ Couns 2014 Jul;96(1):63-71 [FREE Full text] [CrossRef] [Medline]
  69. Ennis L, Rose D, Denis M, Pandit N, Wykes T. Can't surf, won't surf: the digital divide in mental health. J Ment Health 2012 Aug;21(4):395-403 [FREE Full text] [CrossRef] [Medline]
  70. Alnasser A, Sathiaseelan A, Al-Khalifa A, Marais D. Development of 'Twazon': an Arabic app for weight loss. JMIR Res Protoc 2016 May 16;5(2):e76 [FREE Full text] [CrossRef] [Medline]
  71. Alnasser AA, Alkhalifa AS, Sathiaseelan A, Marais D. What overweight women want from a weight loss app: a qualitative study on arabic women. JMIR Mhealth Uhealth 2015;3(2):e41 [FREE Full text] [CrossRef] [Medline]
  72. Alvarado-Martel D, Cañas F, Velasco R, Alcubierre N, López-Ríos L, Rius F, et al. Design, construction, and implementation of an online platform for patients with type 1 diabetes: EncoDiab. Patient Prefer Adherence 2015;9:767-775 [FREE Full text] [CrossRef] [Medline]
  73. Andersen T, Bansler R, Kensing F, Moll J, Nielsen K. Alignment of Concerns: A Design Rationale for Patient Participation in eHealth. In: Proceedings of the 2014 47th Hawaii International Conference on System Sciences. 2014 Presented at: HICSS'14; January 6-9, 2014; Hawaii. [CrossRef]
  74. Andersen T, Bjørn P, Kensing F, Moll J. Designing for collaborative interpretation in telemonitoring: re-introducing patients as diagnostic agents. Int J Med Inform 2011 Aug;80(8):e112-e126. [CrossRef] [Medline]
  75. Armstrong N, Hearnshaw H, Powell J, Dale J. Stakeholder perspectives on the development of a virtual clinic for diabetes care: qualitative study. J Med Internet Res 2007 Aug 9;9(3):e23 [FREE Full text] [CrossRef] [Medline]
  76. Armstrong N, Powell J. Preliminary test of an internet-based diabetes self-management tool. J Telemed Telecare 2008;14(3):114-116. [CrossRef] [Medline]
  77. Armstrong N, Powell J. Patient perspectives on health advice posted on internet discussion boards: a qualitative study. Health Expect 2009 Sep;12(3):313-320. [CrossRef] [Medline]
  78. Atkinson NL, Saperstein SL, Desmond SM, Gold RS, Billing AS, Tian J. Rural eHealth nutrition education for limited-income families: an iterative and user-centered design approach. J Med Internet Res 2009 Jun 22;11(2):e21 [FREE Full text] [CrossRef] [Medline]
  79. Bae J, Wolpin S, Kim E, Lee S, Yoon S, An K. Development of a user-centered health information service system for depressive symptom management. Nurs Health Sci 2009 Jun;11(2):185-193. [CrossRef] [Medline]
  80. Bessell A, Clarke A, Harcourt D, Moss TP, Rumsey N. Incorporating user perspectives in the design of an online intervention tool for people with visible differences: face IT. Behav Cogn Psychother 2010 Oct;38(5):577-596. [CrossRef] [Medline]
  81. Boyd AD, Moores K, Shah V, Sadhu E, Shroff A, Groo V, et al. My Interventional Drug-Eluting Stent Educational App (MyIDEA): patient-centered design methodology. JMIR Mhealth Uhealth 2015;3(3):e74 [FREE Full text] [CrossRef] [Medline]
  82. Buckingham CD, Adams A, Vail L, Kumar A, Ahmed A, Whelan A, et al. Integrating service user and practitioner expertise within a web-based system for collaborative mental-health risk and safety management. Patient Educ Couns 2015 Oct;98(10):1189-1196. [CrossRef] [Medline]
  83. Cade J, Carter M, Wark P, Alwan N, Evans C, Albar S, et al. Development of a UK online 24-hour dietary recall assessment tool: 'My Food 24'. Eur J Epidemiol 2013;28(Supp 1):S50-S51. [CrossRef] [Medline]
  84. Cafazzo JA, Casselman M, Hamming N, Katzman DK, Palmert MR. Design of an mHealth app for the self-management of adolescent type 1 diabetes: a pilot study. J Med Internet Res 2012;14(3):e70 [FREE Full text] [CrossRef] [Medline]
  85. Carter MC, Albar SA, Morris MA, Mulla UZ, Hancock N, Evans CE, myfood24 Consortium. Development of a UK Online 24-h Dietary Assessment Tool: myfood24. Nutrients 2015 Jun;7(6):4016-4032 [FREE Full text] [CrossRef] [Medline]
  86. Cnossen IC, van Uden-Kraan CF, Eerenstein SE, Rinkel R, Aalders IJ, van den Berg K, et al. A participatory design approach to develop a web-based self-care program supporting early rehabilitation among patients after total laryngectomy. Folia Phoniatr Logop 2015;67(4):193-201 [FREE Full text] [CrossRef] [Medline]
  87. Colombo C, Filippini G, Synnot A, Hill S, Guglielmino R, Traversa S, et al. Development and assessment of a website presenting evidence-based information for people with multiple sclerosis: the IN-DEEP project. BMC Neurol 2016 Mar 2;16:30 [FREE Full text] [CrossRef] [Medline]
  88. Coyne I, Prizeman G, Sheehan A, Malone H, While AE. An e-health intervention to support the transition of young people with long-term illnesses to adult healthcare services: design and early use. Patient Educ Couns 2016 Dec;99(9):1496-1504. [CrossRef] [Medline]
  89. Danaher BG, Milgrom J, Seeley JR, Stuart S, Schembri C, Tyler MS, et al. Web-based intervention for postpartum depression: formative research and design of the MomMoodBooster Program. JMIR Res Protoc 2012;1(2):e18 [FREE Full text] [CrossRef] [Medline]
  90. Davies RJ, Nugent CD, Donnelly MP, Hettinga M, Meiland FJ, Moelaert F, et al. A user driven approach to develop a cognitive prosthetic to address the unmet needs of people with mild dementia. Pervasive Mob Comput 2009 Jun;5(3):253-267. [CrossRef]
  91. Dykes PC, Stade D, Chang F, Dalal A, Getty G, Kandala R, et al. Participatory design and development of a patient-centered toolkit to engage hospitalized patients and care partners in their plan of care. AMIA Annu Symp Proc 2014;2014:486-495 [FREE Full text] [Medline]
  92. Enah C, Moneyham L, Vance DE, Gakumo CA, Chandler M. Make it like the real world: adolescents’ recommendations for the design of a digital HIV prevention game. J HIV AIDS Soc Serv 2014 May 28;13(2):163-178. [CrossRef]
  93. Enah C, Piper K, Moneyham L. Qualitative evaluation of the relevance and acceptability of a web-based HIV prevention game for rural adolescents. J Pediatr Nurs 2015 Apr;30(2):321-328. [CrossRef] [Medline]
  94. Fink A, Beck JC. Developing and evaluating a website to guide older adults in their health information searches: A mixed-methods approach. J Appl Gerontol 2015 Aug;34(5):633-651. [CrossRef] [Medline]
  95. Fledderus M, Schreurs KM, Bohlmeijer ET, Vollenbroek-Hutten MM. Development and pilot evaluation of an online relapse-prevention program based on acceptance and commitment therapy for chronic pain patients. JMIR Hum Factors 2015 Jan 5;2(1):e1 [FREE Full text] [CrossRef] [Medline]
  96. Graham ML, Uesugi KH, Niederdeppe J, Gay GK, Olson CM. The theory, development, and implementation of an e-intervention to prevent excessive gestational weight gain: e-Moms Roc. Telemed J E Health 2014 Dec;20(12):1135-1142. [CrossRef] [Medline]
  97. Hearn L, Miller M, Fletcher A. Online healthy lifestyle support in the perinatal period: what do women want and do they use it? Aust J Prim Health 2013;19(4):313-318. [CrossRef] [Medline]
  98. Hearn L, Miller M, Lester L. Reaching perinatal women online: the Healthy You, Healthy Baby website and app. J Obes 2014;2014:573928 [FREE Full text] [CrossRef] [Medline]
  99. Hightow-Weidman LB, Fowler B, Kibe J, McCoy R, Pike E, Calabria M, et al. HealthMpowerment.org: development of a theory-based HIV/STI website for young black MSM. AIDS Educ Prev 2011 Feb;23(1):1-12 [FREE Full text] [CrossRef] [Medline]
  100. Horne B, Newsham A, Velikova G, Liebersbach S, Gilleece M, Wright P. Development and evaluation of a specifically designed website for haematopoietic stem cell transplant patients in Leeds. Eur J Cancer Care (Engl) 2016 May;25(3):402-418. [CrossRef] [Medline]
  101. Kattelmann KK, White AA, Greene GW, Byrd-Bredbenner C, Hoerr SL, Horacek TM, et al. Development of Young Adults Eating and Active for Health (YEAH) internet-based intervention via a community-based participatory research model. J Nutr Educ Behav 2014;46(2):S10-S25. [CrossRef] [Medline]
  102. Kim KK, Logan HC, Young E, Sabee CM. Youth-centered design and usage results of the iN Touch mobile self-management program for overweight/obesity. Pers Ubiquit Comput 2015 Jan;19(1):59-68. [CrossRef]
  103. Kuijpers W, Groen WG, Loos R, Oldenburg HS, Wouters MW, Aaronson NK, et al. An interactive portal to empower cancer survivors: a qualitative study on user expectations. Support Care Cancer 2015 Sep;23(9):2535-2542 [FREE Full text] [CrossRef] [Medline]
  104. Kuijpers W, Groen WG, Oldenburg HS, Wouters MW, Aaronson NK, van Harten WH. Development of MijnAVL, an interactive portal to empower breast and lung cancer survivors: an iterative, multi-stakeholder approach. JMIR Res Protoc 2015 Jan 22;4(1):e14 [FREE Full text] [CrossRef] [Medline]
  105. Lee MK, Park HA, Yun YH, Chang YJ. Development and formative evaluation of a web-based self-management exercise and diet intervention program with tailored motivation and action planning for cancer survivors. JMIR Res Protoc 2013;2(1):e11 [FREE Full text] [CrossRef] [Medline]
  106. Martin-Hammond AM, Abegaz T, Gilbert JE. Designing an over-the-counter consumer decision-making tool for older adults. J Biomed Inform 2015 Oct;57:113-123 [FREE Full text] [CrossRef] [Medline]
  107. Neville C, da Costa D, Mill C, Rochon M, Aviña-Zubieta JA, Pineau CA, et al. The needs of persons with lupus and health care providers: a qualitative study aimed toward the development of the Lupus Interactive Navigator™. Lupus 2014 Feb;23(2):176-182. [CrossRef] [Medline]
  108. Neville C, da Costa D, Rochon M, Peschken CA, Pineau CA, Bernatsky S, et al. Development of the Lupus Interactive Navigator as an empowering web-based eHealth tool to facilitate lupus management: users perspectives on usability and acceptability. JMIR Res Protoc 2016 May 30;5(2):e44 [FREE Full text] [CrossRef] [Medline]
  109. Schlosser D, Campellone T, Kim D, Truong B, Vergani S, Ward C, et al. Feasibility of PRIME: a cognitive neuroscience-informed mobile app intervention to enhance motivated behavior and improve quality of life in recent onset schizophrenia. JMIR Res Protoc 2016 Apr 28;5(2):e77 [FREE Full text] [CrossRef] [Medline]
  110. Siek KA, Khan DU, Ross SE, Haverhals LM, Meyers J, Cali SR. Designing a personal health application for older adults to manage medications: a comprehensive case study. J Med Syst 2011 Oct;35(5):1099-1121. [CrossRef] [Medline]
  111. Todd NJ, Jones SH, Lobban FA. What do service users with bipolar disorder want from a web-based self-management intervention? A qualitative focus group study. Clin Psychol Psychother 2013;20(6):531-543. [CrossRef] [Medline]
  112. Trudeau KJ, Ainscough JL, Trant M, Starker J, Cousineau TM. Identifying the educational needs of menopausal women: a feasibility study. Womens Health Issues 2011;21(2):145-152 [FREE Full text] [CrossRef] [Medline]
  113. Noordegraaf AV, Huirne JA, Pittens CA, van Mechelen W, Broerse JE, Brölmann HA, et al. eHealth program to empower patients in returning to normal activities and work after gynecological surgery: intervention mapping as a useful method for development. J Med Internet Res 2012 Oct 19;14(5):e124 [FREE Full text] [CrossRef] [Medline]
  114. Ward R, Taha KM. Patient involvement as experts in the development and assessment of a smartphone app as a patient education tool for the management of thalassemia and iron overload syndromes. Hemoglobin 2016 Sep;40(5):323-329. [CrossRef] [Medline]
  115. Waterlander W, Whittaker R, McRobbie H, Dorey E, Ball K, Maddison R, et al. Development of an evidence-based mHealth weight management program using a formative research process. JMIR Mhealth Uhealth 2014;2(3):e18 [FREE Full text] [CrossRef] [Medline]
  116. Whitehouse SR, Lam PY, Balka E, McLellan S, Deevska M, Penn D, et al. Co-creation with TickiT: designing and evaluating a clinical eHealth platform for youth. JMIR Res Protoc 2013 Oct 18;2(2):e42 [FREE Full text] [CrossRef] [Medline]
  117. Whittaker R, Maddison R, McRobbie H, Bullen C, Denny S, Dorey E, et al. A multimedia mobile phone-based youth smoking cessation intervention: findings from content development and piloting studies. J Med Internet Res 2008 Nov 25;10(5):e49 [FREE Full text] [CrossRef] [Medline]
  118. Williamson H, Griffiths C, Harcourt D. Developing young person's Face IT: Online psychosocial support for adolescents struggling with conditions or injuries affecting their appearance. Health Psychol Open 2015 Jul;2(2):2055102915619092 [FREE Full text] [CrossRef] [Medline]
  119. Williamson H, Hamlet C, White P, Marques EM, Cadogan J, Perera R, et al. Study protocol of the YP Face IT feasibility study: comparing an online psychosocial intervention versus treatment as usual for adolescents distressed by appearance-altering conditions/injuries. BMJ Open 2016 Dec 3;6(10):e012423 [FREE Full text] [CrossRef] [Medline]
  120. Wright CJ, Dietze PM, Crockett B, Lim MS. Participatory development of MIDY (Mobile Intervention for Drinking in Young people). BMC Public Health 2016;16:184 [FREE Full text] [CrossRef] [Medline]
  121. Wysocki T, Hirschfeld F, Miller L, Izenberg N, Dowshen SA, Taylor A, et al. Consideration of insulin pumps or continuous glucose monitors by adolescents with type 1 diabetes and their parents: stakeholder engagement in the design of web-based decision aids. Diabetes Educ 2016 Dec;42(4):395-407. [CrossRef] [Medline]
  122. Todd NJ, Jones SH, Hart A, Lobban FA. A web-based self-management intervention for bipolar disorder 'Living with Bipolar': a feasibility randomised controlled trial. J Affect Disord 2014 Dec;169:21-29. [CrossRef] [Medline]
  123. Meiland FJ, Bouman AI, Sävenstedt S, Bentvelzen S, Davies RJ, Mulvenna MD, et al. Usability of a new electronic assistive device for community-dwelling persons with mild dementia. Aging Ment Health 2012;16(5):584-591. [CrossRef] [Medline]
  124. Crosby LE, Ware RE, Goldstein A, Walton A, Joffe NE, Vogel C, et al. Development and evaluation of iManage: a self-management app co-designed by adolescents with sickle cell disease. Pediatr Blood Cancer 2017 Jan;64(1):139-145. [CrossRef] [Medline]
  125. Foster C, Calman L, Grimmett C, Breckons M, Cotterell P, Yardley L, et al. Managing fatigue after cancer treatment: development of RESTORE, a web-based resource to support self-management. Psychooncology 2015 Aug;24(8):940-949. [CrossRef] [Medline]
  126. Gordon EJ, Feinglass J, Carney P, Ramirez D, Olivero M, O'Connor K, et al. An interactive, bilingual, culturally targeted website about living kidney donation and transplantation for hispanics: development and formative evaluation. JMIR Res Protoc 2015 Apr 20;4(2):e42 [FREE Full text] [CrossRef] [Medline]
  127. Grant RW, Pandiscio JC, Pajolek H, Woulfe A, Pelletier A, Kvedar J, et al. Implementation of a web-based tool for patient medication self-management: the Medication Self-titration Evaluation Programme (Med-STEP) for blood pressure control. Inform Prim Care 2012;20(1):57-67 [FREE Full text] [CrossRef] [Medline]
  128. Groussard PY, Pigot H, Giroux S. From conception to evaluation of mobile services for people with head injury: a participatory design perspective. Neuropsychol Rehabil 2018 Jul;28(5):667-688. [CrossRef] [Medline]
  129. Hallett J, Maycock B, Kypri K, Howat P, McManus A. Development of a web-based alcohol intervention for university students: processes and challenges. Drug Alcohol Rev 2009 Jan;28(1):31-39. [CrossRef] [Medline]
  130. Heinrich E, Schaper N, de Vries N. Development of the web-based type 2 diabetes education programme: DIEP. Eur Diabetes Nurs 2009 Sep;6(2):51-56. [CrossRef]
  131. McCarthy O, Carswell K, Murray E, Free C, Stevenson F, Bailey JV. What young people want from a sexual health website: design and development of Sexunzipped. J Med Internet Res 2012;14(5):e127 [FREE Full text] [CrossRef] [Medline]
  132. McCrindle RJ, Williams VM, Victor CR, Harvey AP, Nyman SR, Barrett J, et al. Wearable device to assist independent living. Int J Disabil Hum Dev 2011;10(4):349-354. [CrossRef]
  133. Timmerman JG, Tönis TM, Dekker-van Weering MG, Stuiver MM, Wouters MW, van Harten WH, et al. Co-creation of an ICT-supported cancer rehabilitation application for resected lung cancer survivors: design and evaluation. BMC Health Serv Res 2016 Dec 27;16:155 [FREE Full text] [CrossRef] [Medline]
  134. Willems RA, Bolman CA, Mesters I, Kanera IM, Beaulen AA, Lechner L. The Kanker Nazorg Wijzer (Cancer Aftercare Guide) protocol: the systematic development of a web-based computer tailored intervention providing psychosocial and lifestyle support for cancer survivors. BMC Cancer 2015;15:580 [FREE Full text] [CrossRef] [Medline]
  135. Ashurst EJ, Jones RB, Abraham C, Jenner M, Boddy K, Besser RE, et al. The diabetes app challenge: user-led development and piloting of internet applications enabling young people with diabetes to set the focus for their diabetes consultations. Med 2.0 2014 Nov 7;3(2):e5 [FREE Full text] [CrossRef] [Medline]
  136. Badr H, Lipnick D, Diefenbach MA, Posner M, Kotz T, Miles B, et al. Development and usability testing of a web-based self-management intervention for oral cancer survivors and their family caregivers. Eur J Cancer Care (Engl) 2016 Sep;25(5):806-821 [FREE Full text] [CrossRef] [Medline]
  137. Buman MP, Epstein DR, Gutierrez M, Herb C, Hollingshead K, Huberty JL, et al. BeWell24: development and process evaluation of a smartphone. Transl Behav Med 2016 Sep;6(3):438-448 [FREE Full text] [CrossRef] [Medline]
  138. Hallberg I, Taft C, Ranerup A, Bengtsson U, Hoffmann M, Höfer S, et al. Phases in development of an interactive mobile phone-based system to support self-management of hypertension. Integr Blood Press Control 2014;7:19-28 [FREE Full text] [CrossRef] [Medline]
  139. Hong Y, Dahlke DV, Ory M, Hochhalter A, Reynolds J, Purcell NP, et al. Designing iCanFit: a mobile-enabled web application to promote physical activity for older cancer survivors. JMIR Res Protoc 2013;2(1):e12 [FREE Full text] [CrossRef] [Medline]
  140. Jongstra S, Beishuizen C, Andrieu S, Barbera M, van Dorp M, van de Groep B, et al. Development and validation of an interactive internet platform for older people: the Healthy Ageing Through Internet Counselling in the Elderly study. Telemed J E Health 2017 Dec;23(2):96-104. [CrossRef] [Medline]
  141. Shorten A, Fagerlin A, Illuzzi J, Kennedy HP, Lakehomer H, Pettker CM, et al. Developing an internet-based decision aid for women choosing between vaginal birth after cesarean and planned repeat cesarean. J Midwifery Womens Health 2015;60(4):390-400. [CrossRef] [Medline]
  142. Thompson D, Cullen KW, Boushey C, Konzelmann K. Design of a website on nutrition and physical activity for adolescents: results from formative research. J Med Internet Res 2012 Apr 26;14(2):e59 [FREE Full text] [CrossRef] [Medline]
  143. Cullen KW, Thompson D, Boushey C, Konzelmann K, Chen T. Evaluation of a web-based program promoting healthy eating and physical activity for adolescents: teen choice: food and fitness. Health Educ Res 2013 Aug;28(4):704-714 [FREE Full text] [CrossRef] [Medline]
  144. Bandura A. Health promotion by social cognitive means. Health Educ Behav 2004 Apr;31(2):143-164. [CrossRef] [Medline]
  145. Terry DJ, O'Leary JE. The theory of planned behaviour: the effects of perceived behavioural control and self-efficacy. Br J Soc Psychol 1995 Jun;34(Pt 2):199-220. [CrossRef] [Medline]
  146. Prochaska JO, Johnson S, Lee P. The transtheoretical model of behavior change. In: Shumaker SA, Riekart KA, Ockene JK, editors. The Handbook of Health Behavior Change. New York, NY: Springer Publishing Co; 1998:59-84.
  147. Fogg BJ. A behavior model for persuasive design. In: Proceedings of the 4th International Conference on Persuasive Technology. 2009 Presented at: Persuasive'09; April 26-29, 2009; Claremont, CA, USA.
  148. Abras C, Maloney-Krichmar D, Preece J. User-centered design. In: Bainbridge W, editor. Encyclopedia of Human Computer Interaction. Thousand Oaks: Sage Publications; 2004:445-456.
  149. Kidd SA, Kral MJ. Practicing participatory action research. J Couns Psychol 2005;52(2):187-195. [CrossRef]
  150. Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, et al. Framework for design and evaluation of complex interventions to improve health. Br Med J 2000 Sep 16;321(7262):694-696 [FREE Full text] [CrossRef] [Medline]
  151. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. Br Med J 2008;337:a1655 [FREE Full text] [CrossRef] [Medline]
  152. Medical Research Council. 2008. Developing and evaluating complex interventions: new guidance   URL: https://mrc.ukri.org/documents/pdf/complex-interventions-guidance/ [accessed 2019-01-28] [WebCite Cache]
  153. Holmes-Rovner M. International Patient Decision Aid Standards (IPDAS): beyond decision aids to usual design of patient education materials. Health Expect 2007 Jun;10(2):103-107 [FREE Full text] [CrossRef] [Medline]
  154. Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of Web-based and mobile health interventions. J Med Internet Res 2011;13(4):e126 [FREE Full text] [CrossRef] [Medline]
  155. World Health Organization. 2018. Classification of digital health interventions v1.0: a shared language to describe the uses of digital technology for health   URL: http://apps.who.int/iris/handle/10665/260480 [accessed 2018-06-20] [WebCite Cache]
  156. Graffigna G, Barello S, Triberti S. Giving (back) a role to patients in the delivery of healthcare services: theoretical roots of patient engagement. In: Patient Engagement: A Consumer-Centered Model to Innovate Healthcare. Warsaw: De Gruyter Open; 2015:13-24.


ACM: Association for Computing Machinery
CeHRes: Center for eHealth Research and Disease Management
CINAHL: Cumulative Index to Nursing and Allied Health Literature
eHealth: electronic health
HSP: health service providers
HSU: health service users
IEEE: Institute of Electrical and Electronics Engineers
MeSH: Medical Subject Headings
MMAT: Mixed Methods Appraisal Tool
MRC: Medical Research Council
OHP: Optimal Health Program
PAR: participatory action research
UCD: User-Centered Design


Edited by M Benham-Hutchins; submitted 04.07.18; peer-reviewed by L van Velsen, J Apolinário-Hagen; comments to author 09.09.18; revised version received 29.10.18; accepted 10.12.18; published 22.02.19

Copyright

©Gaye Moore, Helen Wilding, Kathleen Gray, David Castle. Originally published in Journal of Participatory Medicine (http://jopm.jmir.org), 22.02.2019.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in Journal of Participatory Medicine, is properly cited. The complete bibliographic information, a link to the original publication on http://jopm.jmir.org, as well as this copyright and license information must be included.