This study uses a multimethod design, drawing on a questionnaire and interviews to collect data from adolescents with type 1 diabetes and secondary diabetes in an outpatient hospital setting. This design allows researchers to gather data at a single point in time, which is suitable for evaluating the psychometric properties of the questions, including internal consistency and construct validity. The study used the COSMIN (Consensus-Based Standards for the Selection of Health Measurement Instruments) [24] checklist (Checklist 1) and the analytical model from Murphy et al [25] study on the Primary Care Outcomes Questionnaire to select measurement instruments and guide reporting on measurement properties. Together, this is characterized by a psychometric approach designed to validate reliability, validity, and sensitivity in measuring user involvement. To establish content validity and identify and correct challenges, we conducted semistructured interviews [26] with adolescents with diabetes. The interviews were used to evaluate the quality of responses and to determine whether the questions elicited the information the author intended. This was combined with a statistical analysis of the 5 PRO questions. These analyses assessed feasibility using the amount and patterns of missing data and construct validity by testing prespecified hypotheses about the correlations between the questions. Through the analyses, we focused on age and gender, the potential effect of age on the pattern and content of participants’ responses, and, finally, whether the validity of the questions depended on age. Consequently, we performed all statistical analyses on the group, stratified by age (≥11‐15 y and >15‐18 y) and gender. In other words, the strength of the interviews is that they provide in-depth information regarding face validity in a narrow sample. In contrast, the strength of the statistical analyses lies in their ability to examine the broader population systematically. During the analytical process, we also involved patient and public (PP) partners.

The interview participants were informed verbally and in writing about the study. Written informed consent was obtained before the interviews. Permission to collect questionnaire data from the medical records was given by the Central Denmark Regional Legal Office (record 1-45-70-21-21). The study was registered in the Central Denmark Regional database (record 663160). All procedures performed were in accordance with the ethical standards of the National Research Committee and the Declaration of Helsinki [27]. The Science and Legal Office, Central Denmark Regional Office, assessed the study 1-16-02-326-19.

The study was conducted among adolescents with type 1 diabetes and secondary diabetes aged 11 to 18 years, with a diabetes duration of more than 1 year. Secondary diabetes refers to diabetes that develops due to another medical condition or as a side effect of certain medications, rather than from autoimmune processes (as in type 1 diabetes). Examples of common causes of secondary diabetes include cystic fibrosis or medical treatment following a transplant. The participants came from 4 pediatric departments in the Central Denmark Region and were recruited between May 2019 and March 2022. All participants had internet access to answer the short questionnaire and could understand, read, and write Danish. For the first interviews, participants were purposively sampled to ensure variation in age and gender. The purposive sampling by gender was chosen because girls’ and boys’ physical, mental, and social development occur at different stages [28]. Convenience sampling was used for the second interview, and 5 prior participants were interviewed again. Participants for the interview came from the outpatient clinic at Steno Diabetes Center Aarhus, and the health care professional assisted in the recruitment.

Before their annual consultation at the clinic, participants completed a short questionnaire assessing symptoms of disordered eating using the Diabetes Eating Problem Survey–Revised (16 items) [29,30], the generic WHO-5 Well-Being Index [19], patients’ overall satisfaction [31], and PRO questions for patient involvement [21]. The short questionnaire was part of standard care and was conducted using AmbuFlex telePRO solutions (Center for Patient-Reported Outcomes), a web-based platform for patients to answer PRO questions. When patients responded to the short questionnaire, their responses were directly accessible in their electronic health record [32]. The PRO questions consist of a 5-item questionnaire measuring the degree of agreement with statements on a 5-point scale, ranging from 0 (“Not at all”) to 5 (“A very high degree”), with an additional option to select “Don’t know” (Textbox 1). Multimedia Appendix 1 outlines the 5 PRO questions and the question concerning patients’ overall satisfaction, which was included in the validation. The question regarding patients’ overall satisfaction is also used in the Danish National Health Survey [31].

Data collection was conducted iteratively and consisted of semistructured interviews and data from the short questionnaire described in the previous subsection.

The semistructured interviews were conducted from December 2020 to October 2021 (20 interviews) and from May 2022 to August 2022 (5 interviews). All 25 interviews were performed by the second author (20 interviews) or the third author (CBA; 5 interviews). The interviews took place in the clinic (n=2), by telephone at home (n=21), or via video (n=2). The interviews were audio-recorded and lasted between 11 and 48 minutes (mean 31 minutes). They were transcribed verbatim. A total of two semistructured interview guides were developed, focusing on (1) how participants constructed their answers, (2) what they interpreted the questions to mean, (3) whether they experienced any difficulties in answering, and (4) other aspects that shed light on the broader circumstances upon which their answers were based [26]. The first interview guide included four topics: (1) practical aspects of completing the short questionnaire (eg, “Tell me about the last time you completed the short questionnaire”), (2) short questionnaire content (eg, “How do you understand this question?”; selected questions from the short questionnaire were read to participants), (3) experiences with assessing their own condition and needs (eg,“What do you think about asking questions regarding the health care professionals’ ability to involve you in your treatment?”), and (4) use of the short questionnaire in consultations (eg, “Tell me about how the short questionnaire was used in the conversation with your doctor/nurse”) (Multimedia Appendix 1). The second interview guide focused solely on the use and value of the 5 PRO questions. These interviews were conducted to gather more data on participants’ experiences with the PRO questions. Participants were asked to express their understanding of all 5 PRO questions and explain what they meant in clinical practice. For example, they were asked what it would mean to “Have had an appropriate number of talks with the health care staff about how I can best handle my illness/condition.” Furthermore, participants were asked to describe whether they found it relevant to ask and use questions regarding adolescent user involvement (Multimedia Appendix 1). During the interviews, think-aloud and cognitive interview techniques were used. These are recommended techniques for investigating content validity [33,34]. The quantitative data for statistical analyses were collected from electronic health records in March 2022, covering participants’ data from May 2019 to February 2021. The data consisted of the 5 PRO questions and 1 question on the patient’s overall satisfaction, which was part of the short questionnaire used in participants’ standard care (described in the “PRO Questions for Patient Involvement and Clinical PRO-Context” section). Furthermore, the data included participants’ age, gender, HbA_1c, treatment, duration of diabetes, and diabetes type.

The analysis of the PRO questions included content validity, feasibility, internal consistency, and construct validity.

Content validity was assessed through an analysis of all the interviews. This was an inductive process using the interpretive description methodology [35]. An important feature of this part of the analysis was the participation of PP partners in the process. The value of including PP partners in research is recognized and recommended, with benefits such as patient perspectives improving both research processes and outcomes to ensure they are patient-centered and focused on relevant patients’ needs [36,37]. The PP partners involved in the analytical process were 2 adolescents (one younger than 15 years and one older than 15 years) with type 1 diabetes. They were recruited from the adolescents’ user panel at the outpatient clinic at Aarhus University Hospital. They had experience with consultations at the outpatient clinic but no experience answering the questionnaires. Multimedia Appendix 2 outlines the PP partners’ participation in the analytical process.

The first part of the content validity analysis consisted of sorting data into broadly coded categories. The categories formed codes related to the 5 PRO questions, participants’ experiences with answering the short and PRO questions, how they valued the PRO questions, and challenges with using them. Further analysis investigated how the different statements were interlinked and highlighted similarities and differences within and between the interviews, looking for patterns, associations, and connections across the data. The first and last authors carried out this initial step. This process led to the first interpretation of the PRO questions’ validity. In this analysis stage, the PP partners were involved (Multimedia Appendix 2). The involvement of PP partners revealed no differences in the interpretation of the data. However, PP partners stressed that even though the participants understood the PRO questions, they preferred to use terms more commonly used by adolescents, for example, replacing “outpatient clinic” with the clinic’s name. After the involvement of PP partners, the results were discussed with the rest of the research team, and final interpretations were made, leading to a description of the dominant characteristics of how participants experienced the use and content of the PRO questions. NVivo (Lumivero) supported the organization, coding, and analytical queries across participants’ characteristics.

The COSMIN checklist guides the statistical analysis of the qualitative data, but some steps were omitted due to data limitations. The main limitation was that there was only 1 question per theoretical item. The statistical analysis focused on how age groups differentiated and was divided into those aged more or less than 15 years. The statistical analysis of structural validity tested 2 hypotheses using the Kendall τ-b correlation coefficient. The difference between the age groups is tested in the proportion missing by a 2-sample test for equality of proportions with continuity correction. The difference between means was tested with the Welch 2-sample 2-tailed t test. All analyses were performed using R version 4.0.2 (R Foundation for Statistical Computing).

Questionnaires from 447 adolescents were included, and 20 of the adolescents participated in the interviews. Participants with secondary diabetes (n=3) were included due to their similar treatment approaches to participants with type 1 diabetes. Participants’ demographic and clinical characteristics are described in Table 1. Compared to the overall demographic population in the Central Denmark Region regarding age, gender, duration of diabetes, diabetes type, and HbA_1c, there are no significant differences [38]. All participants in the interviews reported experience answering questionnaires in school or in a health care setting. They indicated it took 5 to 20 minutes to answer the short questionnaire. Half of the participants had responded to the questionnaires alone, while the rest had support from either parents or a health care professional.

Of the 447 questionnaire responses, 258 (58%) respondents completed all involvement questions, while 189 (42%) were missing at least one of the involvement items. Responses such as “Don’t know” and “Not relevant” were treated as missing in the analyses. The indicators are individually scored from 1 to 5, with higher scores indicating greater involvement. As a total score, they are summarized into a simple summative index and rescaled to range from 0 to 100.

The preliminary analysis shows (Table 2) that each item has a mean above the middle of the scale and an SD of around 1 point. Together, these findings indicate that the scale of the answers is skewed, with responses concentrated at the positive end. It also shows that individuals aged older than 15 years have a significantly higher mean score on 3 of the 5 involvement items and the total index. When grouping by other factors, we observe no significant differences in gender, HbA_1c regulation, or WHO-5 Well-Being Index.

Content validity is assessed through the analysis of the interviews, examining whether respondents understood the questions as intended. Most participants had no or only minor challenges in understanding the PRO questions. The PRO questions “The health care staff encouraged me to ask questions or talk about my concerns” and “I was involved when decisions were made about what was to take place” were understood by all participants. As one participant expressed regarding the first of the questions mentioned above, “Well, it’s like that if they ask you to ask questions or something like that” (ID 11).

In the PRO question, “I have had an appropriate number of talks with the health care staff about how I can best handle my illness/condition,” understanding the term “appropriate” was challenging for 2 participants (ID 1 and ID 2). Additionally, understanding the word “experiences” was challenging for 2 participants (ID 3 and ID 4) in the PRO question, “The health care staff asked questions about my own experiences with my illness/condition.” The PRO question, “I talked to the health care staff about the questions or concerns I had,” was understood by all the participants. However, many of them found it irrelevant to answer, as they described having no questions or concerns. Consequently, they answered, “Not relevant.”

Other words like “concerns” and “health care staff” were not a problem for the participants to understand, as 1 participant explained:

All participants appreciated being asked about their involvement in their care. As 1 participant explained:

Furthermore, being able to complete the questions before their consultation made them reflect more on their care and condition. They expressed that “you must be honest with yourself when answering the questionnaire because if you report a problem, you will be confronted with it during the consultation.” This was viewed positively, but also as a challenge that they believed health care professionals should be aware of:

Since the questionnaire was completed before the consultation, participants had to recall and base their responses on the previous consultation 3 months earlier. Most participants expressed that this was a challenge. When asked by the interviewer, “When you answer the questions, what do you think back on?" 1 participant answered:

As a result, they described that their answers were often based on a more general impression of their clinic consultations.

In the COSMIN framework, feasibility is defined as the ease of application and the availability of the measurement instrument. It is not considered a measurement property and is therefore not assessed using statistical methods; instead, it depends on the specific context of use. In this study, the response rate is used as an indicator of feasibility. Statistical analysis of 447 questionnaires from the short questionnaire showed that for participants aged between ≥11 to 15 years, the proportion of missing responses on the PRO “I talked to the health care staff about the questions or concerns I had” was over 30%, almost 50% higher than for participants aged between >15 and 18 years. This may be interpreted as a sign of difficulty answering the question, especially among the youngest participants.

Internal consistency is defined as the degree of interrelatedness and is assessed using Cronbach α and the interitem correlation, and the mean interitem correlations were calculated as the average of the Pearson correlation coefficients between all item pairs. In both groups, α was well above the recommended threshold of 0.70, with no substantial differences observed between them (Table 2). For both groups, the mean interitem correlation was just below 0.5, which indicates adequate homogeneity among the items without suggesting substantial redundancy.

Construct validity refers to the degree to which scores align with theoretically derived hypotheses, including expected internal relationships, correlations with other instruments, and differences between relevant groups, under the assumption that the instrument validly measures the intended construct. Construct validity was assessed by testing a set of predefined hypotheses. These hypotheses were based on established properties of patient involvement, namely that higher levels of perceived patient involvement are positively associated with greater overall satisfaction and better self-rated health [39]. As expected, the actual Kendall τ-b correlations were strong for overall satisfaction in both age groups. For the WHO-5 Well-Being Index, the picture was different; the correlation was small for the youngest age group and insignificant. For the older group (aged >15‐18 y), the correlation was only in the expected direction but small with significance at the .05 level (Table 3).