Health care systems are facing an era of unprecedented growth in the number of patient-facing eHealth technologies available. Personal health record portals, mobile health (mHealth) apps, clinical videoconferencing platforms, automated texting systems, and other such tools have great potential to reach, engage, and empower patients, to support access to and delivery of care, and to improve outcomes; however, this potential has not yet been widely realized [1-4]. The experiences of many health care organizations underscore that promoting patient adoption and use of these technologies is difficult, and their effective integration into routine care can be elusive. Different patients may have different levels of interest in using eHealth technologies and may choose to abandon use if their expectations are not sufficiently met [5]. Because these technologies have, in many cases, not yet attained widespread recognition or endorsement among health care providers, patient awareness of their availability may be limited [6]. Similarly, the extent to which patient-facing eHealth technologies fit into a patient’s daily life and their larger technological milieu can also directly affect their perceived usefulness and benefits.

Recognizing the importance of these issues, a growing body of work has focused on how best to bolster patient adoption and use of eHealth technologies, monitor their use, and measure their impacts on processes of care and outcomes [2,7,8]. This work, however, has also faced challenges. Particularly, studies that have relied on secondary data (eg, hospital administrative data or technology activity log data) to measure exposure to and use of patient-facing technologies and their effect on health care utilization and outcomes have encountered other issues, the most problematic perhaps being missing data. Many covariates related to technology use (eg, health literacy, education level, income level) are simply not available through such data sources. However, these covariates are essential to understanding technology adoption and adjusting for confounding factors when modeling associations with outcomes. A similar challenge exists for outcomes of interest to health care organizations, including patient-reported outcomes and perceptions of care. These challenges speak to how important it is for health care organizations to consider developing infrastructure capable of offering more complete data and placing the voice of the patient in the foreground as the key stakeholder in efforts to implement and evaluate patient-facing eHealth technologies [2].

The Veterans Health Administration (VHA) has been a pioneering, national leader in developing patient-facing eHealth technologies and integrating their use into clinical practice. Similar to other health care organizations, the VHA has developed a range of such technologies intended for use by the patient population that they serve—veterans of the US military. Table 1 presents key categories of these technologies and a description of each.

In addition to those presented in Table 1, new VHA patient-facing technologies are in continual development. As these technologies have been developed and rolled out at different times and are intended to meet different needs, the extent of their adoption within the veteran population varies considerably [10-12]. The work of developing and implementing these technologies is the responsibility of VHA’s Office of Connected Care, which oversees VHA’s digital health strategy and is focused on improving VHA care through technology that engages veterans beyond traditional health care visits.

Recognizing the need for further insights to advance the implementation and evaluation of their portfolio of technologies, in 2016, the VHA Office of Connected Care, in conjunction with the VHA Quality Enhancement Research Initiative Program, funded an effort to recruit a group of veterans willing to make a long-term commitment to providing feedback on the latest VHA patient-facing technologies and helping the VHA understand their potential benefits. Branded the Veterans Engagement with Technology Collaborative, the overarching goal of this cohort is to directly engage veterans in the evaluation of new VHA patient-facing technologies intended to improve access to care, enhance care coordination, and support self-management, and in so doing, to create new infrastructure to support related research and evaluation activities. Soliciting the expertise of patients and integrating their perspectives into VHA’s technology evaluation efforts demonstrates the values of participatory medicine, a patient-centered philosophy, in action. It also aligns with broader initiatives by VHA and its leadership to enhance veteran engagement in an effort to realize health care system improvements that resonate with the veteran population.

Importantly, the development and use of the Veterans Engagement with Technology Collaborative cohort was designed to meet the criteria for quality improvement and was subsequently reviewed by the Institutional Review Board at the Edith Nourse Rogers Memorial Veterans Hospital in Bedford, Massachusetts, and determined to be such [13]. This designation is an important step toward realizing the National Academy of Medicine’s vision for learning health systems, where “science, informatics, incentives, and culture are aligned for continuous improvement and innovation, with best practices seamlessly embedded in the delivery process [14].”

The specific objectives of the Veterans Engagement with Technology Collaborative cohort were to (1) identify the extent of exposure and use, as well as patient perceptions, of select VHA patient-facing eHealth technologies, (2) understand characteristics and determinants associated with adoption and use of specific VHA patient-facing eHealth technologies, and (3) examine the impact of patient-facing eHealth technology use on select patient-reported outcomes, experience with and perceptions of VHA care.

This paper reports the process by which the Veterans Engagement with Technology Collaborative cohort was developed, describes the baseline cross-sectional survey data collected from cohort members, and details future plans for longitudinal follow-up and cohort maintenance.

The Veterans Engagement with Technology Collaborative cohort is a longitudinal cohort comprised of survey data collection across multiple time-points.

Individuals were eligible for inclusion in the cohort if they were a veteran of the US military and an active user of VHA health care services. In addition, because we aimed to enroll veterans who were users of patient-facing eHealth technology, at the time of screening, all veterans invited to be in the cohort were required to have a mobile phone and have sent at least 5 (but less than 30, which comprised the cut-off for the 95th percentile of secure message volume) secure messages using the patient-to–clinical team secure messaging feature of the VHA personal health record portal in the year prior to recruitment.

The VHA health care system provides health care and other benefits (such as compensation or pension, life insurance, and vocational rehabilitation) to approximately 9.7 million veterans of the US military, which is nearly half of the entire veteran population [15]. On average, compared to civilians, veterans are older, predominantly male, and experience more health concerns, and compared to veterans who do not receive VHA care, veterans who use VHA services tend to be sicker and have less income [16]. The veterans who currently comprise the Veterans Engagement with Technology Collaborative cohort received care from at least 1 of 14 purposefully sampled VHA medical centers across the United States and include residents of most states across both urban and rural settings. In addition to securing a diverse geographic representation of veterans, as well as adequate representation of women and racial or ethnic minority veterans, we selected these 14 facilities based on several criteria deemed important to the overall goals of the Veterans Engagement with Technology Collaborative cohort. Criteria included engaging veterans seen for care at VHA facilities representing different geographic regions of the country that had high rates of adoption of the VHA’s personal health record portal secure messaging feature, a track record of being a site of early adoption for other VHA patient-facing eHealth technologies (eg, VHA’s automated text messaging system, video-to-home telehealth, online scheduling), and plans for implementing other VHA eHealth technologies in the future. Veterans who met these criteria were included on recruitment lists, which detailed their name and contact information (eg, telephone number) as listed in VHA administrative records housed in the VHA Corporate Data Warehouse.

In order to recruit veterans into the cohort, we provided evaluation team members with these recruitment lists, and team members reached out to each veteran by phone to invite them to participate. We called all eligible patients one time unless they requested a call back, in which case, we made one additional follow-up call. During the call, team members read a script to the veterans explaining the purpose of the Veterans Engagement with Technology Collaborative cohort and their eligibility to participate. Evaluation team members also explained the long-term commitment requested of each participating veteran. That is, veterans who consented to participate in the cohort during this recruitment call would be engaged in ongoing survey efforts and periodic evaluation activities over time. If the veteran agreed to participate, the evaluation team member then collected the baseline survey data. Responses were entered in real time, into a secure electronic database system (REDCap, Research Electronic Data Capture; Vanderbilt University). Survey administration took approximately 20 minutes per patient. All recruitment calls and survey data collection efforts occurred in the 2017 calendar year. No incentives were provided at baseline data collection.

We will use the data provided by the cohort to examine novel and important issues related to technology use among veterans, including perceptions of newly developed patient-facing technologies, impacts of use on perceptions of and satisfaction with care delivery, and associations with important health and utilization outcomes (eg, health-related goal setting and attainment, medication adherence, communication with care team members). In this manuscript, we examine frequencies of responses to key survey items. Statistical analyses were conducted using STATA (version 14.2; StataCorp LLC).

We identified and attempted to contact 20,091 veterans who met inclusion criteria for the cohort. Of these veterans, 5877 were reached by phone, 2735 agreed to participate, and 2727 completed the survey (46.4% participation rate).

The veterans who comprised the cohort were 59.9 years old, on average, at the time that the first survey was administered in 2017. Participants were predominantly male (2268/2727, 83.2%), White (2226/2727, 81.6%), and living in their own apartment or house (2519/2696, 93.4%). Most had completed some college (1176/2701, 43.5%) or an advanced degree (1178/2701, 43.6%) and were married or in a civil union (1734/2687, 64.5%). Most (1813/2637, 68.8%) reported that it was not very difficult for them to pay for basics like food and heating or cooling (Table 2).

The most prevalent chronic conditions among this sample were hypertension (1699/2727, 62.3%), osteoarthritis (1444/2727, 53.0%), and depression (1109/2727, 40.7%). Most participants reported that they receive their health care mostly at the VHA (2142/2718, 78.8%), and nearly half (1326/2720, 48.8%) reported living less than 30 minutes away from the VHA at which they received primary care. The majority of the cohort reported being in good (1055/2725, 38.7%) or very good (524/2725, 19.2%) health and that they always take their medications as recommended by their care providers (2273/2698, 84.3%) (Table 3).

The majority of participants reported owning a personal computer (2609/2725, 95.7%), tablet computer (1616/2716, 59.5%), or smartphone (2438/2722, 89.6%). Most (2412/2727, 88.5%) reported that they do not borrow technological devices from others. Most reported that they agree (669/2715, 24.6%) or strongly agree (813/2715, 29.9%) that they are an early adopter of new technology and that they are very comfortable or confident using computers (1878/2705, 69.4%) (Table 4).

Evidence regarding the use and effectiveness of patient-facing technologies is accumulating [26-31], but considerable gaps remain. Given the abundance of new patient-facing technologies that are being (and will continue to be) developed, health care systems will face an ongoing challenge to determine if and how best these technologies can be used to support patients and improve health care quality. The concept of the learning health care system holds that “learning while doing” should be the penultimate goal of health care organizations and emphasizes the importance of appropriate infrastructure, data resources, and partnerships between stakeholders [32,33]. The learning health care system is predicated on the active collaboration among all participants in a system, underscoring how critical it is to engage stakeholders—including patients—in evaluation and implementation efforts [33]. What we have described in this paper is one initiative that is helping to accentuate the voice of the veteran in ongoing efforts to realize the vision of the learning health care system within VHA.

The Veterans Engagement with Technology Collaborative cohort directly engages veterans to understand the potential benefits and possible unintended consequences related to the patient-facing technologies that the VHA is developing and implementing. This new initiative provides a means for veterans and program evaluators to test these technologies on a timeline that more closely reflects their rapid development and evolution. It also supports the rapid evaluation of unexpected but significant changes in the health care system that may influence the role of technology in care delivery. The coronavirus disease 2019 (COVID-19) pandemic and the surge in patient-facing technology use that has accompanied it, is a case in point. The cohort reflects a strong partnership between operational entities within a large health care system and established members of its research and evaluation community to develop new infrastructure to support the broad goal of implementing and measuring the impacts of patient-facing eHealth technologies in practice.

The application and advancement of health- and health care–related technology has the potential to help revolutionize care, improve patient outcomes and satisfaction, and reduce health care costs [34]. Notably, the veterans described in this manuscript have made a commitment to participate in longitudinal follow-up, which will consist of follow-up survey efforts by our team over several years, the content and timing of which will be driven by the VHA’s evaluation needs. Some question items and scales included on the baseline survey will be repeated, thus providing longitudinal data. We expect that other question items and scales will be added based on emergent priorities. Through our follow-up data collection efforts, this cohort of veterans will also serve as a resource to evaluate future technologies, such as new mobile health apps developed for use within the VHA to optimize important outcomes (eg, access to and coordination of services, patient activation and self-management, goal setting and attainment). Cohort member involvement in evaluations is likely to include providing feedback via multiple approaches including targeted surveys and interviews focused on user experiences. Evaluation activities that incorporate the voice of the veteran are increasingly recognized by the VHA health care system and its leadership as powerful approaches to improving health care delivery in ways that reflect the needs and preferences of the veteran population. The Veterans Engagement with Technology Collaborative cohort aligns with other veteran engagement initiatives currently being implemented to improve the policy and patient relevance of VHA research and evaluation activities.

Survey responses are subject to a number of biases (eg, recall bias, response bias), and the cross-sectional design does not allow us to determine causal relationships. Furthermore, the veterans in our sample represent a subset of the veteran population and are known technology users, and in comparison to the general population of veterans who use VHA health care, are approximately one year younger, on average [35], and include a greater proportion of women [36], individuals who are White and of non-Hispanic ethnicity, individuals who report being in fair or poor health [35], and individuals who have health conditions such as depression [37,38], diabetes [39], posttraumatic stress disorder [38,40], hypertension [41], and anxiety [38]. These differences may impact patient-facing eHealth technology preferences or use and limit the generalizability of the data collected thus far, as well as findings from future evaluations conducted with this cohort.

We also acknowledge that the proxy indicators used for technology adoption (ie, veteran use of secure messaging in the prior year, early implementation of eHealth technologies at VHA facilities where veterans receive health care) do not comprehensively reflect all factors that may impact use of patient-facing eHealth technologies. Future work may consider additional factors, for example, aspects of the technology’s design, usability, and utility. In addition, while we recognize the importance of health care providers and their perspectives in the development and evaluation of patient-facing technologies, we have not yet incorporated their perspectives into the initiative. Because all patient-facing technologies have reciprocal repercussions for the health care team members of patients who use them, in the future, we also plan to assess the perspectives of VHA health care team members.

Through the development of the Veterans Engagement with Technology Collaborative cohort, the VHA is laying the foundation for future, rapid, rigorous evaluation and quality improvement efforts that can advance our understanding of the adoption, use, and impact of patient-facing technologies and inform related policy decisions and funding priorities. The development and maintenance of the Veterans Engagement with Technology Collaborative cohort over time establishes a diverse group of veterans who can test emerging VHA patient-facing technologies and technology-based interventions. This infrastructure will help us obtain early feedback on these technologies, as well as advance our understanding of whether certain groups of veterans require extra support to adopt these technologies and use them over time.