Participatory design (PD) is a user-centered design method that actively involves future users as co-designers in generally an iterative design process in which changes to the design based on feedback by users are applied between design sessions [28-30]. This user involvement increases the likelihood that the design fits user needs and will be adopted and used as intended [30]. PD is particularly helpful for developing health information visualizations, that is, text and images with a communicative intent that aim to improve health outcomes by means of improved comprehension, communication, or engagement, and subsequent behavior changes [28]. PD is frequently used in health informatics and has been proven capable of changing clinical practice [28,31]. Since its inception in the late 1970s in Scandinavia in the computer science field, PD has evolved significantly. A recent systematic review showed that users are generally involved throughout a design process in which multiple techniques are used. The techniques can either be predetermined or emergent [29]. Our approach was emergent.

Outcome measures for PD can range (1) development (eg, ideas developed), (2) quality (eg, understandability), and (3) user outcomes of the quality report (eg, intent to prescribe or actual prescriptions) [33]. In this study, the key design decisions that emerged during participatory prototyping are the development outcome measures. The web-based questions in the final feedback meetings (PD technique: stakeholder survey), the comparative content analysis (PD technique: competitive benchmarking), and the HEMAT (PD technique: competitive benchmarking) are the quality outcome measures. A proposal for an effectiveness study has been submitted to assess user outcomes, results of which, if funded, will be reported separately. The proposed study will include both HCPs that are known to have more affinity with health apps (young urban HCPs) and HCPs that are known to be more reluctant (older rural HCPs) and will use widely applicable clinical situations (“vignettes”) to approximate real-life HCP scenarios [18].

The study followed a participatory and emergent research approach conducted within the framework of an EU-funded project. Due to the exploratory and iterative nature of the activities, formal ethics committee approval was not obtained prior to the study. However, ethical considerations were integrated throughout the research process and participant rights were actively protected. Participation in the HCP Advisory Board, the prioritization survey, and final feedback meetings was entirely voluntary, and participants could discontinue participation at any stage. Participants in the HCP Advisory Board confirmed their willingness to participate via email communication and were informed at the beginning of meetings about the purpose of the discussions, the intended use of the collected information, and their right to withdraw or decline to contribute without any consequences. For the survey-based data collection, participants were informed about the purpose of the study and provided consent through the Google Forms interface prior to participation. Participants in the feedback meetings were informed that they could leave the meeting at any time if they did not wish their opinions to be included in the research. The employees of the Dutch Patient Federation who provided feedback signed an informed consent form. The study did not involve vulnerable populations, medical interventions, or the collection of highly sensitive personal data. Data collection focused on professional experiences and perspectives relevant to the project activities. All collected information was handled in accordance with applicable data protection and confidentiality principles within the EU project framework. All members of the HCP Advisory Board received and reviewed the final version of this manuscript and agreed to its publication. See Multimedia Appendix 3 for the reporting guideline for health research involving design [47].

The HCP Advisory Board met 12 times over the course of 18 months (December 8, 2022, to May 29, 2024). The recommended group size of 4-8 [28] was achieved in the iterative meetings, which lasted in between 1 and 1.5 hours. The survey to evaluate the usefulness of the 82304‐2 requirements in decision-making on recommending a health app revealed that in total 37% (30/81) of 82304‐2 requirements were prioritized by at least 50% of the participating 31 HCPs. The medical societies that were contacted to support recruitment, the demographics of the 31 HCPs, and the 30 prioritized 82304‐2 requirements are included in Multimedia Appendix 4. The most prioritized 82304‐2 quality requirement, endorsed by 87% (27/31) of the participating HCPs, was “Peer-reviewed scientific literature used in the development of the app.” The second most prioritized 82304‐2 quality requirements were “Health issues targeted,” “HCPs involved in the development of the app.” “Potential customers and users made aware of health risks,” “Health benefit evidence,” and “Appropriate evidence level for the health benefit,” which were all selected by 80% (25/31) of the participating HCPs. The key design decisions (“development outcomes”) that emerged in the participative prototyping of the 82304‐2 quality report are shown in Textbox 2.

The responses from the HCP Advisory Board to the web-based questions inspired by the TDF (Figure 3) revealed that the HCP Advisory Board members were very positive about the 82304‐2 quality report final prototype: 89% (8/9) agreed or strongly agreed that it was well constructed and 100% (9/9) agreed or strongly agreed that it offered all information and materials that HCPs need to recommend high-quality apps to their patients. In total, 100% (9/9) of the HCP Advisory Board members agreed or strongly agreed to medical societies or organizations having a responsibility to produce guidance on recommending high-quality health apps, while 78% (7/9) agreed or strongly agreed to medical societies having a responsibility in providing training on recommending high-quality health apps. Although the 82304‐2 quality report design was not built with this purpose in mind, 56% (5/9) agreed or strongly agreed that the final prototype nonetheless offered all information and materials for medical societies to produce guidance. The Board members confirmed a perceived need for assistance for medical societies in producing guidance: 100% (9/9) agreed or strongly agreed. Relatedly, 56% (5/9) disagreed with medical societies having clear plans to produce guidance on recommending high-quality apps.

The responses from the medical societies to the web-based questions inspired by the TDF (Figure 4) revealed that the medical societies were also positive about the 82304‐2 quality report final prototype: 88% (7/8) agreed or strongly agreed that it was well constructed. Although the report was not designed with medical societies in mind, 50% (4/8) agreed or strongly agreed that it offered all information and materials for medical societies to create guidance. The results indicated mixed to positive perceptions regarding medical societies’ responsibility to produce guidance on recommending high-quality health apps: 63% (5/8) agreed or strongly agreed and 25% (2/8) slightly disagreed or disagreed. Responses were more reluctant to negative about providing training: 0% (0/8) agreed or strongly agreed, 38% (3/8) slightly agreed, and 38% (3/8) disagreed or strongly disagreed. The results confirmed that medical societies need assistance in developing guidance on recommending high-quality apps to members: 63% (5/8) agreed or strongly agreed. This need was reflected in the negative responses to having clear plans for producing such guidance: 63% (5/8) disagreed or strongly disagreed.

The final remarks, related follow-up to transform the final prototype into the final 82304‐2 quality report design, and the medical societies that participated in the final feedback meetings are included as Multimedia Appendix 5. Two items related to the concept that “an app is not a pill” require further research. These concern a taxonomy of health interventions or health app functionalities and sufficient level of evidence (accepted study design) for different intended uses.

The final FIGMA graphical design software visualization of the design of the 82304‐2 quality report, scrollable and in part clickable to visualize access to the more detailed information, is included in Multimedia Appendix 6. See Figure 5 for screenshots of the FIGMA. The final version of the Word document, which specifies for each of the 82304‐2 quality requirements what information will be included in the 82304‐2 quality report if the requirement is met, not met, or not applicable, is included in Multimedia Appendix 7.

All key design decisions were adhered to (Textbox 3).

The results of the comparative content analysis are included in Multimedia Appendix 8. Except for AppGuide (4 clicks), all reports provided detailed information in at most 2 clicks. The 82304‐2 quality report provides detailed factual information for more quality requirements than DiGA, Digi-HTA, and AppGuide, and with a few exceptions in equal or fewer clicks. Of note, our design effort did not include a directory that provides access to multiple health app quality reports, which might add a click. DiGA differed in having a separate section for HCPs (“Information für Fachkreise”), in not providing (color-coded) quality scores, and in having products either permanently or provisionally listed or not listed anymore, indicating whether funding by statutory health insurance funds is provided. DiGA did provide search options (medical specialty, platform, ICD [International Classification of Diseases] indication, status of the app [permanently included, provisionally, and removed], age [12‐17 years, 18‐65 years, and older than 65 years], gender, language, additional costs, and additional devices [no, included, optional, and necessary]). As the supply of DiGA grows, choosing between DiGA with the same intended use and intended users may become challenging without (color coded) quality scores [48]. To indicate whether individual health app quality requirements were met, AppGuide and 82304‐2 used color coding and standardized text, DiGA-only standardized text, and Digi-HTA descriptive information.

The HEMAT understandability results are included in Multimedia Appendix 9. The 82304‐2 quality report scored highest with 63% (5/8) of items scored yes and 25% (2/8) in part, AppGuide scored second with 57% (4/7) yes and 14% (1/7) in part, Digi-HTA scored 13% (1/8) yes and 25% (2/8) in part, and DiGA scored 13% (1/8) yes and 13% (1/8) in part. Of note, DiGA and 82304‐2 scored best on providing all the information HCPs need, with details including indications and contraindications using ICD-10 (International Classification of Diseases, 10th Revision), functionalities, a standardized granular overview of privacy measures met and not met, and a PICO (Patient or Problem, Intervention, Comparison, Outcome) analysis for clinical evidence. However, DiGA did not score on “excluding information that may distract or overwhelm users unfamiliar with the topic.” AppGuide and 82304‐2 both provide access to detailed information per individual requirement, whereas DiGA and Digi-HTA provide this access to findings for a range of quality requirements within a category. Especially for DiGA, this results in long text documents, which makes finding needed information more challenging. Digi-HTA and AppGuide did not provide this information, or not in the needed granularity. Digi-HTA assesses “suitability of an app for use in healthcare” in general and, as such, does not seem to tailor to the information needs of HCPs in their decision-making for individual patients. AppGuide was distinctive in information on all (322.020) health and wellness apps in the app stores, providing HCPs with a source of information to respond to questions from patients who have come across a health app themselves.

In this study, we used PD techniques to design the 82304‐2 quality report, with an aim to support decision-making on recommending an individual health app to an individual patient by providing HCPs in a user-friendly way the information on the quality of a health app that fits their needs and competences. We managed to engage the Label2Enable HCP Advisory Board, a diverse group of 9 HCPs with expertise in digital health, in participatory prototyping over the course of 18 months, supplemented by 8 medical societies in a final feedback meeting. This resulted in a final 82304‐2 quality report design with detailed information for 90% (27/30) of the 82304‐2 quality requirements HCPs (n=31) perceived of most use value in individual HCP decision-making on recommending a health app. A comparative content analysis and the HEMAT indicated that the quality report design was generally comprehensive and understandable. Final feedback from the 9 HCPs and 8 medical societies was limited, and responses to our web-based questions on the final prototype confirmed promising potential in meeting HCP information needs (“design saturation”). Although the quality report was not designed with the needs of medical societies in creating guidance on recommending high-quality health apps in mind, the quality report design did show potential in also meeting medical societies’ information needs. Feedback from the Dutch Patient Federation indicatively confirmed the usefulness and usability of the quality report for patients. Two items in the quality report related to “an app is not a pill” require further research. These are a taxonomy for health app functional components and, on a more granular intended use level, which study designs are considered adequate evidence.

The recent mapping of HCP information needs and the 82304‐2 quality assessment framework created with the European Society of Cardiology Regulatory Affairs Committee [23] affirms the potential of 82304‐2 to satisfy HCP and potentially also medical society information needs with regard to recommending health apps. More conclusive evaluation of the usefulness and usability of 82304‐2 quality reports by HCPs is recommended, also considering guidance that formal evaluations such as testing comprehension, usability, behavioral intention, or behavior change should follow PD’s preliminary evidence of meeting objectives [28], and the few interventional studies to enhance HCP health app recommendations to date and their limitations [21]. These evaluations should make use of 82304‐2 quality reports with the results of actual app assessments instead of dummy content. The assessed apps should have target users that fit the patient characteristics of the participating HCPs. A particular focus on reluctant and less experienced HCPs should be considered to ensure that the quality report effectively meets the information needs of both HCPs who have and those who have not yet been willing and able to recommend health apps. Both German and French research suggest 3 types of HCPs, being those with positive attitudes toward health apps based on perceived benefits (eg, facilitating and complementing care provision and behavioral effects for patients), those carefully considering benefits (eg, effectiveness and efficiency) and risks (eg, overuse and misuse) for patients and HCPs, and those with negative attitudes (eg, concerns about data privacy, liabilities, actionability of app data, and adverse effects on patient-doctor relationship) [3,16]. Additionally, further research efforts with medical societies and other stakeholders will have to prove the multistakeholder and intercontinental value of the 82304‐2 quality report and related potential in avoiding a duplication of assessment efforts. Such evaluations are expected to enhance the compatibility of the report with the needs of a wide range of HCPs, their medical societies, and with extended efforts potentially more stakeholders that matter to HCPs, such as patients, hospitals, insurers, and HTA bodies. Nontraditional stakeholders (eg, new market entrants such as a telecom company or supermarket that recommends prevention-related apps) may also be considered [49-51].

Medical societies were more ambivalent to negative about their role in providing guidance and training on recommending high-quality health apps than we expected based on medical societies’ common roles in guidance and training and the literature on HCP needs as to medical society guidance with regard to recommending health apps. The perceived need of medical societies for more assistance in creating this type of guidance seems to indicate the lack of clear plans, and perhaps also the ambivalence may be more related to a lack of know-how than a lack of interest. The particularly negative score for training may be due to difficulties designing a training if guidance lacks. Further research and support for medical societies to develop effective guidance and related training seem warranted, ideally without each and every medical society having to invent or reinvent the wheel. We expect that an effort with a limited number of medical societies will reveal that guidance will be only to a limited extent health issue specific. This finding would prove directional to further medical societies and provide a basis for creating guidance and training material (eg, Massive Open Online Courses) for both the medical societies involved and their not directly involved peers. Potential new medical societies such as the German Medical Society of Digital Medicine [52] can be involved in the future—in addition to more industry-guided societies of health app manufacturers.

This study was part of the Horizon Europe Label2Enable project. Another task in this project examined the EU label legislations for energy, food, and medication. All these labels were found to be accompanied by more detailed quality reports. These reports were made available in a database, as were the 3 quality reports we compared with in this study. Although there are many health and wellness apps, their numbers are limited compared with, for example, the number of household appliances in the European Product Registry for Energy Labelling and medical devices in the European Database on Medical Devices. In this research, we created a design for the 82304‐2 quality report. A working scalable version of the 82304‐2 quality report, also referred to as “proof of process” and “proof of production” [44], as well as the related open access database, requires additional funding and yet could build on the learnings with the energy, food, and medication labels, and perhaps even on their existing (EU) platforms or the yet to be established EU database for registration of electronic health record systems and wellness applications described in Article 49 of the European Health Data Space regulation. Having the designated platform connect to routinely used trusted platforms, such as those of medical societies, may be further explored, benefiting from experiences in, for instance, Australia [7], and findings in research in Germany [16,18], where, for example, 73% of diabetes HCPs obtained information on apps from their medical society versus 13% from the DiGA directory and 7% from the national health portal [16].

To our knowledge, this is the first extensive PD study to arrive at a design for a health app quality report with indicative proof of satisfying the information needs of HCPs and potentially other stakeholders, in particular, medical societies and patients. The perceived potential alongside complementary interventions toward making recommending high-quality apps a habit and building HCP confidence in this new skill [26] holds promise in addressing the current lack of high-quality health app recommendations and achieving the related health and societal benefit. Although 82304‐2 is a standard with global applicability, and the 81 global quality requirements were the basis of the 82304‐2 quality report, the information provided including granularity may be specific to information needs of European HCPs. In total, 89% (8/9) of HCPs and all medical and patient societies involved were European. Some of the detailed information in the quality report was adapted to the EU regulatory context (eg, referring to the EU Medical Device Regulation and the [soon to be implemented] European electronic health record exchange format), as was some of the wording (eg, personal data instead of personally identifiable information, and data protection officer instead of person responsible for legal and regulatory compliance of processing of personally identifiable information, following terminology used in the EU General Data Protection Regulation). Differences may also be attributable to health system specifics and context-specific additional quality requirements [53]. We acknowledge that in the aftermath of the COVID-19 pandemic the number of HCPs who evaluated the use value of individual 82304‐2 requirements (n=31) as well as the number of participants in the final feedback meetings (n=17) was limited, and that the HCP Advisory Board consisted largely of HCPs with experience in the field and interest in the topic, who were also involved in the participatory prototyping. This likely had an effect on the responses to our questions based on the TDF with regard to the quality report being well constructed and offering the information that HCPs need. On the other hand, the 8 medical societies that participated in the final feedback meetings were not involved in this design effort and agreed with the quality report being well constructed. We recognize that our current work concerned prototyping and not real-life deployment. We recommend more conclusive evaluation of usefulness, usability, and use of the 82304‐2 quality report by HCPs in a demonstrator phase or effectiveness study, with particular effort to also include reluctant and less experienced HCPs and other relevant stakeholders. This phase would also enable next stages of the design process, progressing from “proof of concept” and “proof of product” to “proof of process” and “proof of production” [44], beyond the current indicative assessments of ORCHA lead app assessors as to feasibility to produce and scale production of the 82304‐2 quality report.

We succeeded in our aim to design the 82304‐2 quality report including its information detail and found promising potential for its distinctive usefulness for HCPs in recommending health apps, complementing the 82304‐2 quality label for which an effect on willingness to recommend apps was previously found. Additionally, initial feedback on the potential compatibility of the quality report for medical societies in creating guidance and for patients in considering a health app was positive. Further efforts are needed to test and fine-tune the HCP, multistakeholder and intercontinental usefulness, usability, and use of the 82304‐2 quality report; to support medical societies in providing guidance and training; and to evolve from the current FIGMA design to a working scalable version of the 82304‐2 quality report and related open access database.