This study followed a participatory, exploratory, sequential, and mixed methods design (Figure 1) [40-43]. Exploratory designs are commonly used for developing new instruments or measures [43-45] and are particularly useful when a culturally responsive instrument needs to be created [43].

The original MMI process was implemented in the 2016-2017 admissions cycle [15], and a timeline of the program procedures and revisions can be found in Figure 2. The MMI encounters that are a part of our admissions process are delivered and evaluated by our simulations center team which includes simulations specialists and SPs. These encounters require no medical knowledge by the applicant and can simulate communication with a patient, peer, or coworker. For each MMI encounter, the applicant interacts with a single SP while being observed through a 2-way window by another independent SP in real time. At the conclusion of the encounter, the observing SP and performing SP collaboratively discuss the applicant’s performance and skills and complete the rubric to provide a quantitative outcome. The evaluation by 2 SPs helps eliminate potential bias, ensures accuracy in scoring, and streamlines the fast pace of MMIs. Furthermore, a standardized rubric and 2 SPs ensure accuracy and timely completion in the limited time allotted between encounters.

SPs, simulation center staff, and members of the admissions committee participated in an evaluation of the MMI program after the 2017-2018 cycle. This evaluation included a review of the MMI rubric and admissions committee documents for quality improvement purposes. According to evaluation results, partnering with SPs to cocreate an addition to the evaluation rubric could improve the MMI program.

For the SPs involved in the MMI during the medical school admissions process, the simulation specialists and researchers hosted a workshop in August 2018 to allow for the cocreation and improvement of the rubric. The workshop included a short didactic session and active participation by the SPs. During the didactic presentation (Multimedia Appendix 1), the researchers gave a short presentation to (1) review the rubric used in the previous year, (2) present reasons for creating the new question, (3) explain the basic survey design [46], and (4) discuss the importance of stakeholder engagement in this type of work.

Upon completing the presentation, the SPs were divided into small groups and given approximately 30 minutes to create 1 question and the corresponding responses. The created questions were written and displayed on large pieces of paper placed on the walls and on 2 screens in the room. Subsequently, everyone was reassembled into a large group to discuss the developed questions. Voting and further discussion led to a single SP-created question. Once the question had been finalized, the SPs were then divided into 2 groups to write response options for the newly created question. Collaborative discussion, voting, and editing led to a co-designed set of response options.

The quantitative data were collected from various sources, including 1084 medical school applications, MMI process forms, and admission status for 3 cohorts: (1) 2018-2019, (2) 2019-2020, and (3) 2020-2021. Applicants’ demographics were gathered from school applications. Averages of the 4 SP encounters for each MMI category were provided for individual applicants. Admission status was categorized into 3 categories: accepted, declined, or wait-listed (reviewed applicants awaiting consideration from those who withdrew their application prior to acceptance). The wait-listed applicants were excluded from the regression models. In all, 3 applicants did not have admissions or matriculations status listed and were excluded from analysis.

Quantitative data analyses were performed using SPSS software (version 26; IBM Corp). Descriptive statistics included frequencies, means and SDs, and percentages. The internal consistency reliability of the MMI was measured using a Cronbach α test evaluating all 10 variables within the MMI: (1) opening the encounter, (2) empathy, (3) nonverbal behavior, (4) verbal behavior, (5) listens well, (6) therapeutic relationship, (7) negotiation of the plan, (8) closing the encounter, (9) rank of readiness, and (10) admissions committee member. To determine if the SPs may be influenced by the demographics of the applicants, the applicants’ age, sex, and underrepresented in medicine (URM) status (dichotomous yes/no) were regressed (multiple linear regression) on the average MMI score (excluding the admissions committee member score). Finally, we sought to determine if medical school admissions status could be predicted based upon the variables within the SP-scored MMI (9 variables). A forward stepwise binary logistic regression was used to identify possible predictors of acceptance status (accepted or declined) out of the following MMI candidate variables: opening the encounter, empathy, nonverbal behavior, verbal behavior, listens well, therapeutic relationship, negotiation of the plan, closing the encounter, and rank of readiness. The forward logistic regression used the likelihood-ratio test to enter or remove variables from the model.

Ethical oversight of the project was conducted by the Institutional Review Board at the University of South Carolina (Pro00069266).

Feedback from the SPs, simulation center staff, and admissions department revealed that the checklist rubric (Multimedia Appendix 2) was easy for SPs to use and for the admissions committee to integrate into their decision-making process. In contrast, some SPs would write extensive qualitative comments about applicants, whereas others would leave few or no notes on those they interviewed. Additionally, it was revealed that the admissions committee was not using these written comments by the SPs in their applicant deliberations. Therefore, adding a single MMI rubric item that evaluates applicants based on their interactions with SPs during the scenario might reduce the SPs’ need to write qualitative comments and provide additional information for the admissions committee. To this end, a workshop was planned and implemented to develop a question and responses to provide a global assessment of a medical school applicant.

In attendance at the workshop were 3 simulation specialists, 2 researchers, and 21 SPs. The mean age of the SPs was 53.1 (SD 12.43; range 27-70) years, and a majority were female (16/21, 76%) and had a bachelor’s degree or higher (18/21, 86%). The 6 SP-created questions are listed in Table 1. The SPs identified that 3 of the 6 questions were dichotomous in nature and concluded that if selected, these 3 questions would not be a useful global measure assessment tool. Additionally, they identified certain words that they deemed to be either useful or problematic, such as “successful,” “characteristics,” “1st year” (vs medical student in general), “ability,” and “readiness.” Several SPs deliberated over the ways to strengthen the questions. For example, the SPs suggested adding “Based on this interaction,” to the beginning of the second question to help the person answering the question focus on the interaction with the applicant and not the applicant’s qualities overall. Additionally, the question of whether communication skills could be developed or if they are innate and unchangeable was discussed.

At the end of the discussion, voting resulted in questions 6 and 4 receiving 10 and 9 votes, respectively (2 of the 21 SPs had to leave the workshop before the voting process). Additional discussion led to a “tie-breaking vote,” yielding question 6 as the top choice. The SPs, however, determined that both questions had some aspects they wanted to emphasize. Thus, a single question was merged from both, which resulted in the combined global question: “Based on the candidate’s communication and interpersonal skills, rate this candidate’s readiness for medical school.”

As shown in Table 2, both groups developed response options for the global readiness question. There was a brief discussion before a vote was taken, and rubric 2 was chosen with some minor group edits. The bottom 2 scale items were modified to add readiness in light of the question. The word adequate was replaced by the word proficient in the third scale item. On the advanced scale item, the words following the slash may be listed interchangeably instead of in the specific order listed.

In this study, we demonstrated that SPs were able to develop a useful, credible, and relevant measure that can help the medical school admissions committee evaluate applicants beyond academic achievements. Although the MMI is only 1 portion of data used in a holistic review of medical school applicants, it can be useful to examine the impact and process of the MMI in the medical school admissions. Additionally, the created global readiness question that was incorporated into the MMI process contributed to selecting candidates for medical school.

Beginning with the 2018-2019 application cycle, simulation specialists integrated the revised rubric with the readiness question into training and conducting the MMI. This process was successfully adopted by the SPs involved in the MMI. As the SPs began using the new method of quantitatively scoring the applicant’s readiness, they found this 1 question to be more efficient than providing qualitative feedback for each applicant. Since there are only about 3 minutes between applicant encounters, the readiness question allows SPs to provide more concise feedback without being rushed.

Including stakeholders in the development of surveys and assessment instruments has been useful in other settings outside of simulation centers and has led to credible and relevant tools [47-50]. In this setting, working with SPs to create the rank of readiness question revealed what could be appropriately assessed through the lens of the SPs. The creation of the question through patient and public involvement yielded a more objective and standardized measure for scoring purposes. Additionally, stakeholder involvement contributed to a sense of value and co-ownership in the admissions process. Furthermore, by codeveloping this rubric item with the SPs, we built on their previous MMI skills, allowed item development to be iterative, allowed SPs to own the process, and accommodated their scoring needs. Incorporating these and other processes have been found to be associated with higher rates of positive research outcomes in cocreated projects [28,29].

The results of the workshop positively impact the simulation specialists providing a standardized scoring opportunity for all applicants. Training for standardization is critical in this role. Due to the collaborative nature of the workshop, the simulation specialists were able to emphasize the value of the SPs’ voices and consistent use of the evaluation tool, which is consistent with findings in other settings [27,51]. Moreover, the involvement of the SPs in the development of the question facilitates coaching of new SPs by simulation specialists to use this question to validate or summarize the data from the other rubric sections.

Our results showed that the MMI overall and the readiness question specifically were able to predict medical school acceptance. This finding is consistent with previous findings showing that SPs are able to assess students’ communication skills [52-55]. Furthermore, our findings are consistent with previous results showing that the SPs were unbiased in their assessment of the interviewees [10,56]. The observed interviewees’ demographics (age, sex, and race) predicted only 3% of the SPs’ MMI overall score. Although the findings were statistically significant, the predicted amount of change within the different demographic variables was relatively small. There is a possibility that bias was reduced because 2 SPs scored each interviewee collaboratively.

Although the MMI rank of readiness score was statistically significant for predicting an applicant’s acceptance to medical school overall and for the 2018-2019 and 2019-2020 cohorts, this was not the case for the 2020-2021 cohort. One potential explanation for this lack of significance is that the entire cycle was conducted in a web-based format due to the COVID-19 pandemic. The rubric was not changed to reflect the web-based interview environment, which could potentially account for some of the differences. For example, students were automatically given points for elements that could not be performed in a web-based environment (eg, knocking on door prior to entering). Additionally, nonverbal behavior was challenging to assess, as illustrated by score differences between cohorts with in-person or web-based MMIs (Multimedia Appendix 2). Although medical schools have faced many challenges in moving to a web-based interview format, it has been found that web-based interviews can be as reliable as traditional interviews in making sound decisions on applicants [57]. More research is needed to assess the impacts of the COVID-19 pandemic on medical school admissions and web-based SPs encounters.

Despite the SPs’ ability to create a measure, several other factors play a role in the admissions selection process beyond the MMI including, but not limited to, grade point average, Medical College Admission Test scores, recommendation letters, personal statements, and distance traveled (overcoming adversity or obstacles). These additional factors were not included in the models. Although the SP portion of the MMI excelled at predicting acceptance, specificity was low overall and for each cohort. Other elements of the application may have a greater impact on the applicants who were declined. Although the actual results of the statistical analysis might not be generalizable to other settings, lessons learned from the process for having patient and public involvement when creating a measurement instrument can benefit other institutions.

The standard rubric used for MMI may need to be revised to remove items that cannot be performed over the web (eg, knocking on door, appropriate touch, and sustaining personal space) for admission years when web-based MMI is the standard. This potential, revised rubric could also be used to assess learners during telehealth encounters, and the new rubric would need to be evaluated. Additionally, the MMI rubric without the final readiness question is currently being used as the communication rubric for assessing learners during standard SP encounters and Objective Structured Clinical Examinations. The global question is specifically intended for SPs to determine applicant readiness for medical school, so it should not be used for students who have matriculated. However, it could be beneficial to replicate the workshop with SPs to identify a global question for the communication rubric. Finally, an additional investigation could be conducted to see how matriculating students’ MMI scores compare with the future communication scores throughout their time in medical school.

The collaborative nature of this project between stakeholders, including nonacademics and researchers, was vital for the success of this project. This study shows that SPs bring a critical perspective that can improve the admissions process of evaluating medical school applicants through the MMI process. They also can further incorporate themselves as team members by cocreating an effective global question to improve the evaluation of the applicants.