Published on in Vol 12, No 3 (2020): Jul-Sep

Preprints (earlier versions) of this paper are available at, first published .
Supporting the Implementation of Connected Care Technologies in the Veterans Health Administration: Cross-Sectional Survey Findings from the Veterans Engagement with Technology Collaborative (VET-C) Cohort

Supporting the Implementation of Connected Care Technologies in the Veterans Health Administration: Cross-Sectional Survey Findings from the Veterans Engagement with Technology Collaborative (VET-C) Cohort

Supporting the Implementation of Connected Care Technologies in the Veterans Health Administration: Cross-Sectional Survey Findings from the Veterans Engagement with Technology Collaborative (VET-C) Cohort

Original Paper

1eHealth Partnered Evaluation Initiative, Edith Nourse Rogers Memorial Veterans Hospital, Bedford, MA, United States

2Center of Innovation for Complex Chronic Healthcare, Hines Veterans Affairs Hospital, Hines, IL, United States

3Division of Health Informatics and Implementation Science, Department of Population and Quantitative Health Sciences, University of Massachusetts Medical School, Worcester, MA, United States

4Northwestern University Feinberg School of Medicine, Chicago, IL, United States

5Center for Healthcare Organization and Implementation Research, Edith Nourse Rogers Memorial Veterans Hospital, Bedford, MA, United States

6Department of Health Law, Policy, and Management, Boston University School of Public Health, Boston, MA, United States

7Office of Connected Care, Veterans Health Administration, US Department of Veterans Affairs, Washington, DC, United States

8Department of Population and Data Sciences, University of Texas Southwestern Medical Center, Dallas, TX, United States

Corresponding Author:

Bella Etingen, PhD

Center of Innovation for Complex Chronic Healthcare

Hines Veterans Affairs Hospital

5000 5th Ave

Hines, IL, 60141

United States

Phone: 1 708 202 4922


Background: Widespread adoption, use, and integration of patient-facing technologies into the workflow of health care systems has been slow, thus limiting the realization of their potential. A growing body of work has focused on how best to promote adoption and use of these technologies and measure their impacts on processes of care and outcomes. This body of work currently suffers from limitations (eg, cross-sectional analyses, limited patient-generated data linked with clinical records) and would benefit from institutional infrastructure to enhance available data and integrate the voice of the patient into implementation and evaluation efforts.

Objective: The Veterans Health Administration (VHA) has launched an initiative called the Veterans Engagement with Technology Collaborative cohort to directly address these challenges. This paper reports the process by which the cohort was developed and describes the baseline data being collected from cohort members. The overarching goal of the Veterans Engagement with Technology Collaborative cohort is to directly engage veterans in the evaluation of new VHA patient-facing technologies and in so doing, to create new infrastructure to support related quality improvement and evaluation activities.

Methods: Inclusion criteria for veterans to be eligible for membership in the cohort included being an active user of VHA health care services, having a mobile phone, and being an established user of existing VHA patient-facing technologies as represented by use of the secure messaging feature of VHA’s patient portal. Between 2017 and 2018, we recruited veterans who met these criteria and administered a survey to them over the telephone.

Results: The majority of participants (N=2727) were male (2268/2727, 83.2%), White (2226/2727, 81.6%), living in their own apartment or house (2519/2696, 93.4%), and had completed some college (1176/2701, 43.5%) or an advanced degree (1178/2701, 43.6%). Cohort members were 59.9 years old, on average. The majority self-reported their health status as being good (1055/2725, 38.7%) or very good (524/2725, 19.2%). Most cohort members owned a personal computer (2609/2725, 95.7%), tablet computer (1616/2716, 59.5%), and/or smartphone (2438/2722, 89.6%).

Conclusions: The Veterans Engagement with Technology Collaborative cohort is an example of a VHA learning health care system initiative designed to support the data-driven implementation of patient-facing technologies into practice and measurement of their impacts. With this initiative, VHA is building capacity for future, rapid, rigorous evaluation and quality improvement efforts to enhance understanding of the adoption, use, and impact of patient-facing technologies.

J Participat Med 2020;12(3):e21214




Health care systems are facing an era of unprecedented growth in the number of patient-facing eHealth technologies available. Personal health record portals, mobile health (mHealth) apps, clinical videoconferencing platforms, automated texting systems, and other such tools have great potential to reach, engage, and empower patients, to support access to and delivery of care, and to improve outcomes; however, this potential has not yet been widely realized [1-4]. The experiences of many health care organizations underscore that promoting patient adoption and use of these technologies is difficult, and their effective integration into routine care can be elusive. Different patients may have different levels of interest in using eHealth technologies and may choose to abandon use if their expectations are not sufficiently met [5]. Because these technologies have, in many cases, not yet attained widespread recognition or endorsement among health care providers, patient awareness of their availability may be limited [6]. Similarly, the extent to which patient-facing eHealth technologies fit into a patient’s daily life and their larger technological milieu can also directly affect their perceived usefulness and benefits.

Recognizing the importance of these issues, a growing body of work has focused on how best to bolster patient adoption and use of eHealth technologies, monitor their use, and measure their impacts on processes of care and outcomes [2,7,8]. This work, however, has also faced challenges. Particularly, studies that have relied on secondary data (eg, hospital administrative data or technology activity log data) to measure exposure to and use of patient-facing technologies and their effect on health care utilization and outcomes have encountered other issues, the most problematic perhaps being missing data. Many covariates related to technology use (eg, health literacy, education level, income level) are simply not available through such data sources. However, these covariates are essential to understanding technology adoption and adjusting for confounding factors when modeling associations with outcomes. A similar challenge exists for outcomes of interest to health care organizations, including patient-reported outcomes and perceptions of care. These challenges speak to how important it is for health care organizations to consider developing infrastructure capable of offering more complete data and placing the voice of the patient in the foreground as the key stakeholder in efforts to implement and evaluate patient-facing eHealth technologies [2].

Transforming Care Through eHealth Technologies in the Veterans Health Administration

The Veterans Health Administration (VHA) has been a pioneering, national leader in developing patient-facing eHealth technologies and integrating their use into clinical practice. Similar to other health care organizations, the VHA has developed a range of such technologies intended for use by the patient population that they serve—veterans of the US military. Table 1 presents key categories of these technologies and a description of each.

Table 1. Select VHA patient-facing eHealth technologies [9].
Key telehealth services

Remote patient monitoringHealth data (eg, blood pressure, weight, glucose level) is gathered by a device issued to the patient and that data is, in turn, sent to the patient’s care team.

Video appointmentsCamera and audio on smartphones, computers, tablets, and other devices are used to support a video appointment between a patient and their care team from the comfort of the patient’s home.
Personal health record portal

VHA’sa tethered patient portal offers health education resources and features that support transactions with the health care system (eg, medication refilling), communication between a patient and their VHA clinical team members (eg, asynchronous secure messaging), self-management support (eg, tracking behaviors and symptoms), and access to the content of the patient’s medical record (eg, Blue Button).
Automated text-messaging

VHA’s protocol-driven automated text-messaging system provides tailored support for condition-specific self-management and other health behaviors through one-way and two-way messaging.
Mobile apps

VHA has developed a suite of mobile apps (which are designed to address the unique needs of the Veteran population) intended to promote wellness and healthy behaviors, provide condition-specific self-management support, support other transactions with the health care system, and enhance clinical management.

aVHA: Veterans Health Administration.

In addition to those presented in Table 1, new VHA patient-facing technologies are in continual development. As these technologies have been developed and rolled out at different times and are intended to meet different needs, the extent of their adoption within the veteran population varies considerably [10-12]. The work of developing and implementing these technologies is the responsibility of VHA’s Office of Connected Care, which oversees VHA’s digital health strategy and is focused on improving VHA care through technology that engages veterans beyond traditional health care visits.

Recognizing the need for further insights to advance the implementation and evaluation of their portfolio of technologies, in 2016, the VHA Office of Connected Care, in conjunction with the VHA Quality Enhancement Research Initiative Program, funded an effort to recruit a group of veterans willing to make a long-term commitment to providing feedback on the latest VHA patient-facing technologies and helping the VHA understand their potential benefits. Branded the Veterans Engagement with Technology Collaborative, the overarching goal of this cohort is to directly engage veterans in the evaluation of new VHA patient-facing technologies intended to improve access to care, enhance care coordination, and support self-management, and in so doing, to create new infrastructure to support related research and evaluation activities. Soliciting the expertise of patients and integrating their perspectives into VHA’s technology evaluation efforts demonstrates the values of participatory medicine, a patient-centered philosophy, in action. It also aligns with broader initiatives by VHA and its leadership to enhance veteran engagement in an effort to realize health care system improvements that resonate with the veteran population.

Importantly, the development and use of the Veterans Engagement with Technology Collaborative cohort was designed to meet the criteria for quality improvement and was subsequently reviewed by the Institutional Review Board at the Edith Nourse Rogers Memorial Veterans Hospital in Bedford, Massachusetts, and determined to be such [13]. This designation is an important step toward realizing the National Academy of Medicine’s vision for learning health systems, where “science, informatics, incentives, and culture are aligned for continuous improvement and innovation, with best practices seamlessly embedded in the delivery process [14].”


The specific objectives of the Veterans Engagement with Technology Collaborative cohort were to (1) identify the extent of exposure and use, as well as patient perceptions, of select VHA patient-facing eHealth technologies, (2) understand characteristics and determinants associated with adoption and use of specific VHA patient-facing eHealth technologies, and (3) examine the impact of patient-facing eHealth technology use on select patient-reported outcomes, experience with and perceptions of VHA care.

This paper reports the process by which the Veterans Engagement with Technology Collaborative cohort was developed, describes the baseline cross-sectional survey data collected from cohort members, and details future plans for longitudinal follow-up and cohort maintenance.


The Veterans Engagement with Technology Collaborative cohort is a longitudinal cohort comprised of survey data collection across multiple time-points.


Individuals were eligible for inclusion in the cohort if they were a veteran of the US military and an active user of VHA health care services. In addition, because we aimed to enroll veterans who were users of patient-facing eHealth technology, at the time of screening, all veterans invited to be in the cohort were required to have a mobile phone and have sent at least 5 (but less than 30, which comprised the cut-off for the 95th percentile of secure message volume) secure messages using the patient-to–clinical team secure messaging feature of the VHA personal health record portal in the year prior to recruitment.


The VHA health care system provides health care and other benefits (such as compensation or pension, life insurance, and vocational rehabilitation) to approximately 9.7 million veterans of the US military, which is nearly half of the entire veteran population [15]. On average, compared to civilians, veterans are older, predominantly male, and experience more health concerns, and compared to veterans who do not receive VHA care, veterans who use VHA services tend to be sicker and have less income [16]. The veterans who currently comprise the Veterans Engagement with Technology Collaborative cohort received care from at least 1 of 14 purposefully sampled VHA medical centers across the United States and include residents of most states across both urban and rural settings. In addition to securing a diverse geographic representation of veterans, as well as adequate representation of women and racial or ethnic minority veterans, we selected these 14 facilities based on several criteria deemed important to the overall goals of the Veterans Engagement with Technology Collaborative cohort. Criteria included engaging veterans seen for care at VHA facilities representing different geographic regions of the country that had high rates of adoption of the VHA’s personal health record portal secure messaging feature, a track record of being a site of early adoption for other VHA patient-facing eHealth technologies (eg, VHA’s automated text messaging system, video-to-home telehealth, online scheduling), and plans for implementing other VHA eHealth technologies in the future. Veterans who met these criteria were included on recruitment lists, which detailed their name and contact information (eg, telephone number) as listed in VHA administrative records housed in the VHA Corporate Data Warehouse.


In order to recruit veterans into the cohort, we provided evaluation team members with these recruitment lists, and team members reached out to each veteran by phone to invite them to participate. We called all eligible patients one time unless they requested a call back, in which case, we made one additional follow-up call. During the call, team members read a script to the veterans explaining the purpose of the Veterans Engagement with Technology Collaborative cohort and their eligibility to participate. Evaluation team members also explained the long-term commitment requested of each participating veteran. That is, veterans who consented to participate in the cohort during this recruitment call would be engaged in ongoing survey efforts and periodic evaluation activities over time. If the veteran agreed to participate, the evaluation team member then collected the baseline survey data. Responses were entered in real time, into a secure electronic database system (REDCap, Research Electronic Data Capture; Vanderbilt University). Survey administration took approximately 20 minutes per patient. All recruitment calls and survey data collection efforts occurred in the 2017 calendar year. No incentives were provided at baseline data collection.

Baseline Data Collection

Survey Measures

The Veterans Engagement with Technology Collaborative baseline survey was used to gather patient-level information that is not readily available in clinical or administrative databases, such as patient perceptions of access to and current use of health care and patient-facing technologies, perceptions of health care team member support of these technologies, patient-provider communication, and sociodemographic information such as health literacy and financial status.

We asked participants to report on factors associated with their health and health care use, including whether they usually receive health care from the VHA, from outside of the VHA, or both, and how long it takes to travel from their home to their VHA primary care doctor’s office. We also asked them whether anyone helps them manage their health or health care, how they perceive their overall health status [17], and the extent to which they adhere to taking their prescribed medications [18].

In addition, several questions addressed health-related goal setting behaviors; specifically, participants were asked, “In the last 6 months, did anyone in your VHA provider’s office talk with you about specific goals for your health?” and “In the past 6 months, have you set any goals related to your health?” Those who set a goal were asked, “What health-related goal or goals have you made in the past 6 months?”; “Have you been able to achieve this health-related goal?”; and “Have you used an app on a smartphone or tablet to help you achieve/work on this health-related goal?”

Technology Ownership and Use

Technology ownership and use questions assessed personal computer ownership, tablet computer ownership, mobile phone ownership, whether participants ever borrow any of these devices from others, and whether participants ever use devices to measure and send health information (eg, blood pressure, blood glucose level, weight) to their care providers. We also asked participants whether they like to be among the first to get a new device, tech gadget, or app when it comes out (ie, do they consider themselves early adopters), whether they use social media (ie, Twitter, Facebook, Instagram, Pinterest) [19], and how comfortable or confident they feel using computers on a scale of 0 (not at all) to 5 (very) [20].

We used adapted items from the VHA Survey of Healthcare Experience of Patients to assess participant perceptions of their health and health care–related communication, including how easy it is for them to communicate with their care providers when needed on a scale of 1 (very easy) to 5 (very difficult), how often (in the prior 6 months) they received a response within 1 day when they needed to communicate with their care provider’s office on a scale of 1 (never) to 4 (always), and how often (in the prior 6 months) they received a response from their care provider’s office as soon as they needed it when they contacted the office after hours on a scale of 1 (never) to 4 (always).

We also asked participants how big of a problem on a scale of 1 (very big problem) to 5 (not a problem) each of the following are for them: poor communication between different doctors or clinics, disagreements between their doctors about their diagnosis or the best treatment for them, and having their concerns ignored or overlooked by their health care providers. In addition, we asked participants to report how confident they feel filling out medical forms by themselves on a scale of 1 (not at all) to 5 (extremely) [21], and how easy or hard they find it to understand medical statistics on a scale of 1 (very easy) to 4 (very hard) [22].

Additionally, we asked participants to report their agreement on a scale of 1 (strongly disagree) to 7 (strongly agree) with a number of statements about secure messaging: their health care team encourages them to ask questions using secure messaging; in secure messaging, their health care team answers their questions related to their health fully and carefully; and the secure messages they receive make them feel that their health care team cares about them as a person.

We collected demographic information including age, gender, race, ethnicity, relationship status, highest level of education achieved, living arrangement, and financial difficulty.

Data Linked From VHA Records

We obtained data on chronic health conditions and information used to calculate a Hierarchical Condition Community [23,24] score for the veterans in our cohort from the VHA Corporate Data Warehouse. Hierarchical Condition Community scores represent a comorbidity index that takes into account an individual’s age, gender, medical diagnoses, and eligibility for Medicare and Medicaid services [23,24]. Typically, the range of Hierarchical Condition Community scores is between 0.9 and 1.7; scores less than 1 are often interpreted as healthy [25].

Statistical Analyses

We will use the data provided by the cohort to examine novel and important issues related to technology use among veterans, including perceptions of newly developed patient-facing technologies, impacts of use on perceptions of and satisfaction with care delivery, and associations with important health and utilization outcomes (eg, health-related goal setting and attainment, medication adherence, communication with care team members). In this manuscript, we examine frequencies of responses to key survey items. Statistical analyses were conducted using STATA (version 14.2; StataCorp LLC).

Responses to survey items intended to characterize the sample and gather information on covariates are presented below in the narrative and accompanying tables.

Response Rate and Cohort Derivation

We identified and attempted to contact 20,091 veterans who met inclusion criteria for the cohort. Of these veterans, 5877 were reached by phone, 2735 agreed to participate, and 2727 completed the survey (46.4% participation rate).


The veterans who comprised the cohort were 59.9 years old, on average, at the time that the first survey was administered in 2017. Participants were predominantly male (2268/2727, 83.2%), White (2226/2727, 81.6%), and living in their own apartment or house (2519/2696, 93.4%). Most had completed some college (1176/2701, 43.5%) or an advanced degree (1178/2701, 43.6%) and were married or in a civil union (1734/2687, 64.5%). Most (1813/2637, 68.8%) reported that it was not very difficult for them to pay for basics like food and heating or cooling (Table 2).

Table 2. Demographics.
Age (years) (N=2727)

mean (SD)59.9 (13.1)

Gender (N=2727), n (%)

Male2268 (83.2)

Female459 (16.8)
Race (N=2727), n (%)

White2226 (81.6)

Black or African American317 (11.6)

Asian13 (0.5)

Native Hawaiian or other Pacific Islander6 (0.2)

American Indian or Alaskan Native63 (2.3)

Other86 (3.2)

Declined to answer56 (2.1)
Ethnicity (N=2727), n (%)

Yes, Hispanic or Latino135 (5.0)

No, not Hispanic or Latino2592 (95.1)
Relationship status (n=2687), n (%)

Married or in a civil union1734 (64.5)

Neither married, nor in a civil uniona953 (35.5)
Education status (n=2701), n (%)

High school graduate or less347 (12.9)

At least some college or vocational school (1-4 years)1176 (43.5)

Master’s, professional, or doctoral degree1178 (43.6)
Living arrangement (n=2696), n (%)

Own apartment or house2519 (93.4)

Friend or relative’s apartment or house120 (4.5)

Otherb57 (2.1)
Financial difficultyc (n=2637), n (%)

Not very hard1813 (68.8)

Somewhat hard, hard, or very hard824 (31.3)

aDefined as engaged or in a relationship, single, separated, divorced, or widowed.

bDefined as school or dormitory, hospital or detox center, nursing home or assisted living, car or street, or jail or prison.

cBased on response to the question “How hard is it for you (and your family) to pay for the very basics like food and heating/cooling?”

Health and Health Care Use

The most prevalent chronic conditions among this sample were hypertension (1699/2727, 62.3%), osteoarthritis (1444/2727, 53.0%), and depression (1109/2727, 40.7%). Most participants reported that they receive their health care mostly at the VHA (2142/2718, 78.8%), and nearly half (1326/2720, 48.8%) reported living less than 30 minutes away from the VHA at which they received primary care. The majority of the cohort reported being in good (1055/2725, 38.7%) or very good (524/2725, 19.2%) health and that they always take their medications as recommended by their care providers (2273/2698, 84.3%) (Table 3).

Table 3. Health and health care use.
Chronic conditionsa (N=2727), n (%)

Acute myocardial infarction297 (10.9)

Atrial fibrillation227 (8.3)

Heart failure206 (7.6)

Ischemic heart disease646 (23.7)

Peripheral vascular disease352 (12.9)

Hypertension1699 (62.3)

Asthma575 (21.1)

Breast cancer23 (0.8)

Colorectal cancer35 (1.3)

Prostate cancer105 (3.9)

Lung cancer32 (1.2)

Endometrial cancer2 (0.1)

Chronic kidney disease646 (23.7)

Chronic obstructive pulmonary disease105 (3.9)

Depression1109 (40.7)

Diabetes1024 (37.6)

Osteoarthritis1444 (53.0)

Stroke160 (5.9)

Posttraumatic stress disorder773 (28.4)

Anxiety705 (25.9)

Traumatic brain injury205 (7.5)
Health status (general) (n=2725), n (%)

Excellent144 (5.3)

Very good524 (19.2)

Good1055 (38.7)

Fair797 (29.3)

Poor205 (7.5)
Hierarchical Condition Community scorea (N=2727), n (%)

Mean (SD)0.3 (0.03)

Health care receipt (n=2718), n (%)

Mostly at the VHAb2142 (78.8)

Mostly outside VHA159 (5.9)

About half in VHA, half outside VHA417 (15.3)
Travel time (to VHA primary care doctor’s office) (n=2720), n (%)

<30 minutes1326 (48.8)

31 to 60 minutes970 (35.7)

>60 minutes424 (15.6)
Assistance in managing health or health care (N=2727), n (%)

Paid caregiver92 (3.4)

Spouse/partner694 (25.5)

Children157 (5.8)

Family or extended family member176 (6.5)

Friend100 (3.7)

Other29 (1.1)

N/Ac1717 (63.0)
Medication adherenced (n=2698), n (%)

All (100%) of the time2273 (84.3)

Not all of the time425 (15.8)

aIn the prior five years.

bVHA: Veterans Health Administration.

cN/A: not applicable.

dBased on response to the question “In the past month, how often did you take your medications as the doctor prescribed?”

Technology Ownership and Use

The majority of participants reported owning a personal computer (2609/2725, 95.7%), tablet computer (1616/2716, 59.5%), or smartphone (2438/2722, 89.6%). Most (2412/2727, 88.5%) reported that they do not borrow technological devices from others. Most reported that they agree (669/2715, 24.6%) or strongly agree (813/2715, 29.9%) that they are an early adopter of new technology and that they are very comfortable or confident using computers (1878/2705, 69.4%) (Table 4).

Table 4. Technology ownership and use.
Owns a desktop or laptop computer (n=2725), n (%)

Yes2609 (95.7)

No116 (4.3)
Owns a tablet computer (iPad, Kindle Fire, etc) (n=2716), n (%)

Yes1616 (59.5)

No1100 (40.5)
Mobile phone ownershipa (n=2722), n (%)

Smartphoneb2438 (89.6)

Nonsmartphone mobile phone261 (9.6)

No mobile phone23 (0.8)
Borrow devices from others (N=2727), n (%)

Sometimes use friend’s device48 (1.8)

Sometimes use family member’s device167 (6.1)

Use device at work77 (2.8)

Sometimes use library/senior center/hospital/other location’s device42 (1.5)

No2412 (88.5)
Use of devices to measure and send health measurements to health care team (n=2717), n (%)

Yes679 (25.0)

No2038 (75.0)
Early tech adopter (n=2715), n (%)

Strongly agree813 (29.9)

Agree669 (24.6)

Neutral692 (25.5)

Disagree323 (11.9)

Strongly disagree218 (8.0)
Social media use (N=2727), n (%)

Twitter552 (20.2)

Facebook1955 (71.7)

Instagram466 (17.1)

Pinterest479 (17.6)
Comfort or confidence using computers (n=2705), n (%)

Very comfortable or confident1878 (69.4)

Less than very comfortable or confident827 (30.6)

aBased on response to the statement “If you have multiple cell phones, select the one you use most often.”

biPhone, Android, Blackberry, Windows phone, Symbian, or some other type of smartphone.

Veterans Engagement With Technology Collaborative Cohort and the Learning Health Care System

Evidence regarding the use and effectiveness of patient-facing technologies is accumulating [26-31], but considerable gaps remain. Given the abundance of new patient-facing technologies that are being (and will continue to be) developed, health care systems will face an ongoing challenge to determine if and how best these technologies can be used to support patients and improve health care quality. The concept of the learning health care system holds that “learning while doing” should be the penultimate goal of health care organizations and emphasizes the importance of appropriate infrastructure, data resources, and partnerships between stakeholders [32,33]. The learning health care system is predicated on the active collaboration among all participants in a system, underscoring how critical it is to engage stakeholders—including patients—in evaluation and implementation efforts [33]. What we have described in this paper is one initiative that is helping to accentuate the voice of the veteran in ongoing efforts to realize the vision of the learning health care system within VHA.

The Veterans Engagement with Technology Collaborative cohort directly engages veterans to understand the potential benefits and possible unintended consequences related to the patient-facing technologies that the VHA is developing and implementing. This new initiative provides a means for veterans and program evaluators to test these technologies on a timeline that more closely reflects their rapid development and evolution. It also supports the rapid evaluation of unexpected but significant changes in the health care system that may influence the role of technology in care delivery. The coronavirus disease 2019 (COVID-19) pandemic and the surge in patient-facing technology use that has accompanied it, is a case in point. The cohort reflects a strong partnership between operational entities within a large health care system and established members of its research and evaluation community to develop new infrastructure to support the broad goal of implementing and measuring the impacts of patient-facing eHealth technologies in practice.

The application and advancement of health- and health care–related technology has the potential to help revolutionize care, improve patient outcomes and satisfaction, and reduce health care costs [34]. Notably, the veterans described in this manuscript have made a commitment to participate in longitudinal follow-up, which will consist of follow-up survey efforts by our team over several years, the content and timing of which will be driven by the VHA’s evaluation needs. Some question items and scales included on the baseline survey will be repeated, thus providing longitudinal data. We expect that other question items and scales will be added based on emergent priorities. Through our follow-up data collection efforts, this cohort of veterans will also serve as a resource to evaluate future technologies, such as new mobile health apps developed for use within the VHA to optimize important outcomes (eg, access to and coordination of services, patient activation and self-management, goal setting and attainment). Cohort member involvement in evaluations is likely to include providing feedback via multiple approaches including targeted surveys and interviews focused on user experiences. Evaluation activities that incorporate the voice of the veteran are increasingly recognized by the VHA health care system and its leadership as powerful approaches to improving health care delivery in ways that reflect the needs and preferences of the veteran population. The Veterans Engagement with Technology Collaborative cohort aligns with other veteran engagement initiatives currently being implemented to improve the policy and patient relevance of VHA research and evaluation activities.


Survey responses are subject to a number of biases (eg, recall bias, response bias), and the cross-sectional design does not allow us to determine causal relationships. Furthermore, the veterans in our sample represent a subset of the veteran population and are known technology users, and in comparison to the general population of veterans who use VHA health care, are approximately one year younger, on average [35], and include a greater proportion of women [36], individuals who are White and of non-Hispanic ethnicity, individuals who report being in fair or poor health [35], and individuals who have health conditions such as depression [37,38], diabetes [39], posttraumatic stress disorder [38,40], hypertension [41], and anxiety [38]. These differences may impact patient-facing eHealth technology preferences or use and limit the generalizability of the data collected thus far, as well as findings from future evaluations conducted with this cohort.

We also acknowledge that the proxy indicators used for technology adoption (ie, veteran use of secure messaging in the prior year, early implementation of eHealth technologies at VHA facilities where veterans receive health care) do not comprehensively reflect all factors that may impact use of patient-facing eHealth technologies. Future work may consider additional factors, for example, aspects of the technology’s design, usability, and utility. In addition, while we recognize the importance of health care providers and their perspectives in the development and evaluation of patient-facing technologies, we have not yet incorporated their perspectives into the initiative. Because all patient-facing technologies have reciprocal repercussions for the health care team members of patients who use them, in the future, we also plan to assess the perspectives of VHA health care team members.


Through the development of the Veterans Engagement with Technology Collaborative cohort, the VHA is laying the foundation for future, rapid, rigorous evaluation and quality improvement efforts that can advance our understanding of the adoption, use, and impact of patient-facing technologies and inform related policy decisions and funding priorities. The development and maintenance of the Veterans Engagement with Technology Collaborative cohort over time establishes a diverse group of veterans who can test emerging VHA patient-facing technologies and technology-based interventions. This infrastructure will help us obtain early feedback on these technologies, as well as advance our understanding of whether certain groups of veterans require extra support to adopt these technologies and use them over time.


This work was supported by the US Department of Veterans Affairs, Office of Connected Care, and Office of Research and Development, Health Services Research and Development Service, Quality Enhancement Research Initiative Program (PEC 15-470; principal investigator: TPH). The views expressed in this article are those of the authors and do not necessarily reflect the position and/or policy of the Department of Veterans Affairs or of the United States Government.

Conflicts of Interest

None declared.

  1. Tulu B, Trudel J, Strong DM, Johnson SA, Sundaresan D, Garber L. Patient Portals: An Underused Resource for Improving Patient Engagement. Chest 2016 Jan;149(1):272-277. [CrossRef]
  2. Nazi KM, Turvey CL, Klein DM, Hogan TP. A Decade of Veteran Voices: Examining Patient Portal Enhancements Through the Lens of User-Centered Design. J Med Internet Res 2018 Jul 10;20(7):e10413 [FREE Full text] [CrossRef] [Medline]
  3. Hennemann S, Beutel ME, Zwerenz R. Ready for eHealth? Health Professionals' Acceptance and Adoption of eHealth Interventions in Inpatient Routine Care. J Health Commun 2017 Mar;22(3):274-284. [CrossRef] [Medline]
  4. Waldren SE, Agresta T, Wilkes T. Technology Tools and Trends for Better Patient Care: Beyond the EHR. Fam Pract Manag 2017;24(5):28-32 [FREE Full text] [Medline]
  5. Eysenbach G. The Law of Attrition. J Med Internet Res 2005 Mar;7(1):e11 [FREE Full text] [CrossRef] [Medline]
  6. Black H, Gonzalez R, Priolo C, Schapira MM, Sonnad SS, Hanson CW, et al. True "Meaningful Use": Technology Meets Both Patient and Provider Needs. Am J Manag Care 2015 May 01;21(5):e329-e337 [FREE Full text] [Medline]
  7. Sieverink F, Kelders SM, van Gemert-Pijnen JE. Clarifying the Concept of Adherence to eHealth Technology: Systematic Review on When Usage Becomes Adherence. J Med Internet Res 2017 Dec 06;19(12):e402 [FREE Full text] [CrossRef] [Medline]
  8. Yeager CM, Benight CC. If We Build It, Will They Come? Issues of Engagement with Digital Health Interventions for Trauma Recovery. Mhealth 2018;4:37 [FREE Full text] [CrossRef] [Medline]
  9. US Department of Veterans Affairs. Office of Connected Care Website. 2018.   URL: [accessed 2020-08-11] [WebCite Cache]
  10. US Department of Veterans Affairs. VA Telehealth Services Fact Sheet.   URL: [accessed 2020-08-14]
  11. US Department of Veterans Affairs. My HealtheVet Metrics 2019.   URL: [accessed 2020-08-14]
  12. US Department of Veterans Affairs. Mobile Behavior Design Lab.   URL: [accessed 2020-08-14]
  13. US Department of Veterans Affairs. VA Handbook 1058.05. VA Operations Activities That May Constitute Research 2011.
  14. National Academy of Medicine. The Learning Health System Series.   URL: [accessed 2020-08-14]
  15. US Department of Veterans Affairs. VA Utilization Profile FY 2016.   URL: [accessed 2020-08-14]
  16. Eibner C, Krull H, Brown KM, Cefalu M, Mulcahy AW, Pollard M, et al. Current and Projected Characteristics and Unique Health Care Needs of the Patient Population Served by the Department of Veterans Affairs. Rand Health Q 2016 May 09;5(4):13 [FREE Full text] [Medline]
  17. RAND Corporation. 36-Item Short Form Survey (SF-36).   URL: [accessed 2020-08-14]
  18. Gehi AK, Ali S, Na B, Whooley MA. Self-Reported Medication Adherence and Cardiovascular Events in Patients With Stable Coronary Heart Disease: The Heart and Soul Study. Arch Intern Med 2007 Sep 10;167(16):1798-1803. [CrossRef]
  19. Greenwood S, Perrin A, Duggan M. 2016 Social Media Update, PEW Research Center.   URL: https:/​/www.​​internet/​wp-content/​uploads/​sites/​9/​2016/​11/​PI_2016.​11.​11_Social-Media-Update_FINAL.​pdf [accessed 2020-08-14]
  20. Cho AH, Arar NH, Edelman DE, Hartwell PH, Oddone EZ, Yancy WS. Do Diabetic Veterans Use the Internet? Self-Reported Usage, Skills, and Interest in Using My HealtheVet Web Portal. Telemed J E Health 2010 Jun;16(5):595-602. [CrossRef] [Medline]
  21. Powers BJ, Trinh JV, Bosworth HB. Can This Patient Read and Understand Written Health Information? JAMA 2010 Jul 07;304(1):76-84. [CrossRef]
  22. Woloshin S, Schwartz LM, Welch HG. Patients and Medical Statistics. Interest, Confidence, and Ability. J Gen Intern Med 2005 Nov;20(11):996-1000. [CrossRef]
  23. Pope GC, Kautter J, Ellis RP, Ash AS, Ayanian JZ, Lezzoni LI, et al. Risk Adjustment of Medicare Capitation Payments Using the CMS-HCC Model. Health Care Financ Rev 2004;25(4):119-141 [FREE Full text] [Medline]
  24. Morgan RO, Petersen LA, Hasche JC, Davila JA, Byrne MM, Osemene NI, et al. VHA Pharmacy Use in Veterans with Medicare Drug Coverage. Am J Manag Care 2009 Mar 16;15(3):e1-e8 [FREE Full text] [Medline]
  25. Yeatts JP, Sangvai D. HCC Coding, Risk Adjustment, and Physician Income: What You Need to Know. Fam Pract Manag 2016;23(5):24-27 [FREE Full text] [Medline]
  26. Restuccia JD, Cohen AB, Horwitt JN, Shwartz M. Hospital Implementation of Health Information Technology and Quality of Care: Are They Related? BMC Med Inform Decis Mak 2012 Sep 27;12(1):109. [CrossRef]
  27. Weigel FK, Switaj TL, Hamilton J. Leveraging Health Information Technology to Improve Quality in Federal Healthcare. US Army Med Dep J 2015:68-74. [Medline]
  28. Frimpong JA, Jackson BE, Stewart LM, Singh KP, Rivers PA, Bae S. Health Information Technology Capacity at Federally Qualified Health Centers: A Mechanism for Improving Quality of Care. BMC Health Serv Res 2013 Jan 31;13(1):35. [CrossRef]
  29. Or CKL, Tao D. Does the Use of Consumer Health Information Technology Improve Outcomes in the Patient Self-Management of Diabetes? A Meta-Analysis and Narrative Review of Randomized Controlled Trials. Int J Med Inform 2014 May;83(5):320-329. [CrossRef] [Medline]
  30. Wang J, Wang Y, Wei C, Yao NA, Yuan A, Shan Y, et al. Smartphone Interventions for Long-Term Health Management of Chronic Diseases: An Integrative Review. Telemed J E Health 2014 Jun;20(6):570-583. [CrossRef] [Medline]
  31. Buhi ER, Trudnak TE, Martinasek MP, Oberne AB, Fuhrmann HJ, McDermott RJ. Mobile Phone-Based Behavioural Interventions for Health: A Systematic Review. Health Educ J 2012 Jul 10;72(5):564-583. [CrossRef]
  32. Slutsky JR. Moving Closer To A Rapid-Learning Health Care System. Health Aff (Millwood) 2007 Jan;26(2):w122-w124. [CrossRef]
  33. Forrest CB, Margolis P, Seid M, Colletti RB. PEDSnet: How A Prototype Pediatric Learning Health System Is Being Expanded Into A National Network. Health Aff (Millwood) 2014 Jul;33(7):1171-1177. [CrossRef]
  34. Klein S, Hostetter M, McCarthy D. A Vision for Using Digital Health Technologies to Empower Consumers and Transform the US Health Care System. The Commonwealth Fund 2014. [CrossRef]
  35. Huang G, Kim S, Muz B, Gasper J. 2017 Survey of Veteran Enrollees’ Health and Use of Health Care.   URL: [accessed 2020-08-14]
  36. National Center for Veterans Analysis and Statistics. VA Utilization Profile FY 2017.   URL: [accessed 2020-08-14]
  37. VA Office of Research and Development. VA Research on Depression.   URL: [accessed 2020-08-14]
  38. Trivedi RB, Post EP, Sun H, Pomerantz A, Saxon AJ, Piette JD, et al. Prevalence, Comorbidity, and Prognosis of Mental Health Among US Veterans. Am J Public Health 2015 Dec;105(12):2564-2569. [CrossRef]
  39. VA Office of Research and Development. VA Research on Diabetes.   URL: [accessed 2020-08-14]
  40. VA National Center for PTSD. How Common is PTSD in Veterans?.   URL: [accessed 2020-08-14]
  41. Yu W, Ravelo A, Wagner TH, Phibbs CS, Bhandari A, Chen S, et al. Prevalence and Costs of Chronic Conditions in the VA Health Care System. Med Care Res Rev 2003;60(3_Suppl):146S-167S. [CrossRef]

mHealth: mobile health
VHA: Veterans Health Administration

Edited by S Woods; submitted 08.06.20; peer-reviewed by S Jang, D Klein; comments to author 26.06.20; revised version received 30.06.20; accepted 05.08.20; published 30.09.20


©Bella Etingen, Daniel J Amante, Rachael N Martinez, Bridget M Smith, Stephanie L Shimada, Lorilei Richardson, Angela Patterson, Thomas K Houston, Kathleen L Frisbee, Timothy P Hogan. Originally published in Journal of Participatory Medicine (, 30.09.2020.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in Journal of Participatory Medicine, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.