Revalidating “Mini Surgical Theater Educational Environment Measure (mini-STEEM)”: Is it a Content Valid Instrument?

Jahangeer Ahmad; Hadia Baig; Zaitoon Zafar; Talat Waseem

Abstract

Introduction: Operating Room (OR) based learning experience has traditionally remained challenging owing to complex medical, psychosocial, educational, and administrative factors, and so has its measurement. Recently, mini-STEEM, an abbreviated form of Surgical Theater Educational Experience Measure (STEEM), has been employed to evaluate medical students’ OR-based learning experience. This study aims to assess its content and construct validity in light of an updated extensive literature review.

Methods: Extensive literature review was done to identify and evaluate various factors affecting students' OR-based learning within the OR-setting. Expert validation of the items was done in light of constructs identified through literature review by 8 experts who had experience both in surgical teaching and medical education. The item content validity index (I-CVI) was assessed. Thematic analysis of the qualitative responses was done to understand the dynamics of validity scores. Factor analysis was performed to see the factor loadings for the items and their model fitness.

Results: Against previously identified 26 constructs, mini-STEEM items were expert-validated for clarity, relevance, and comprehension. The item content validity index was suboptimal and did not cover all of the dimensions and constructs identified through a previous literature review. Reliability was checked through Cronbach’s alpha (0.779) after piloting through a cohort of students (n=156). Qualitative analysis revealed that mini-STEEM does not cover all of the domains essential for student learning within the OR setting, and does not have adequate representative items to address all elements. Exploratory factor analysis (EFA) showed reasonable loadings; however, the model fitness through confirmatory factor analysis (CFA) was not possible considering low or no representation of most of the themes.

Conclusion: In light of evolving literature evidence, the content validity of mini-STEEM to assess the quality of medical students' OR-based learning experience remains low, despite having good reliability and construct loadings.

Revalidating “Mini Surgical Theater Educational Environment Measure (mini-STEEM)”: Is it a Content Valid Instrument?

PRACTICE POINTS:

Operating Room-based student learning is multifactorial and has traditionally been suboptimal.

Medical students’ perspective about OR-based learning differs from a surgical resident's perspective, and many factors have been previously identified to affect students' OR-based learning.

Recently, a growing body of literature has emerged and has been collated to emphasize at least five domains that govern the student learning experience within the OR setting. These include the quality of the structured learning process, organizational support, psychosocial perspectives, and factors related to teachers and the student.

Mini-Surgical Theater Educational Experience Measure (STEEM) or mini-STEEM was previously designed to evaluate students' experience; however, in light of the evolving body of literature, mini-STEEM may no longer be a content valid instrument.

Introduction

The dynamic environment within the Operating Room (OR) provides a real-life venue for medical students’ learning1 2. However, there are many medical, social, ethical, and administrative factors involved, which make learning within the OR setting exigent for teachers and learners both1 2 3. Recently, many experts in this area have generated a new conceptual framework highlighting various important factors that influence the quality of OR-based student learning1 2. Lyon's model focuses on student-driven learning within the OR, while Roberts et al. stress a more deliberate learning process for the medical students1 4. There is a consensus that the learning process within at large remains opportunistic, unstructured, un-driven, and hence substandard1 2. Consequently, the OR setting's standardized learning process and quality assurance remain the key for enhanced OR learning.

Nagraj et al. previously proposed the instrument ‘mini-STEEM’ to assess the quality of student learning within the OR environment, which is an abbreviated form of Kevin Cassar’s Surgical Theater Educational Environment Measure5 6. This instrument by Kevin Cassar was primarily designed for the residents and is a reliable and content valid6. Recently in multiple systematic literature reviews, many important OR-based learning determinants have been identified, and a more comprehensive conceptual framework has been developed 1 2 7. Mini-STEEM, although reliable with reasonable Cronbach’s alpha value, does not assess all domains which pertain to medical students' learning experience within the OR setting7.

In light of an improved understanding of OR-based learning and multiple recent literature reviews that identify many essential domains of student learning within the OR setting, there is a clear need for re-validation of the mini-STEEM. This study aims to assess the content validity, reliability, and construct validity of mini-STEEM.

Methodology

Following ethical approval from the local institutional review, a board study was conducted, as shown in Figure 1. In the first phase, we relied on the previously published literature evidence1 2 7. The literature search was conducted on PubMed, ERIC, and Google Scholar. Thematic analysis and review were performed to identify the factors influencing OR-based student learning. Additionally, the study consisted of the analysis of various models and scales currently being used to assess the OR-based student learning1 2 8 9 10 11 12. The conceptual framework and previously identified themes were used as reference (Table 1 and Figure 2)7.

Table 1: Expert Validation of mini-STEEM based on Conceptual Framework

Themes (5)

Subthemes (26)

Mini-STEEM Items

Representative Qualitative Statements

I-CVI

STRUCTURED LEARNING PROCESS

(Structured Course /Lesson Planning, Content Selection, Delivery, Assessment & Administration)

Structured or Opportunistic OR learning

No representative Item

“The students prefer a structured and well-planned learning process within operating theatre realm instead of opportunistic learning process—this inventory does not adequately assess whether learning is structured or opportunistic”

“Like any learning process learning objectives should be formulated prior to lesson and following the convention of medical education should be specific, measurable, attainable, relevant, targeted & time-bound (SMART)…these learning objectives should conform to the rest of teaching…… students should have room for their own personal learning objectives to promote diversity in learning process—Mini-STEEM does not address these important themes”

“The items pertaining to content selection are quite relevant, however they do not represent all of factors influencing this process. For example, it does not assess the process of curricular design and the involvement of students in this whole process. It does assess the ‘briefing’ component of Robert’s “Briefing-Intraoperative Teaching-Debriefing” model however does not appraise the mechanism of intraoperative teaching and debriefing session.”

“Mini-STEEM overlooks and does not evaluate the process of assessment or delivery of constructive feedback to the student about their performance in OR learning”

“The instrument does not gauge the process of assessment of skills and affective domains of learning.”

“In mini-STEEM, theatre sessions reflect a segment of opportunity for learning. The items however have been primarily designed for the residents and it reflects in structure of these items. For example, it may be quite hard to provide opportunity to scrub along with the surgeon and make this a standard considering the increasing number of students and residents. The last three items clearly pertain to residents not medical students.”

Design and Communication of Learning Objectives for OR learning

No representative Item

Clarity of Learning Objectives

No representative Item

Feasibility of learning objectives to be realistically achievable

No representative Item

Synchronization of the learning objectives with rest of the teaching.

No representative Item

Importance of Personal Learning Objectives in OR Learning

No representative Item

Content Selection, delivery, assessment and their alignment to essential aspects of graduate learning

Before the operation my trainer discusses the surgical technique planned

The elective operating list has the right case mix to suit my training

The variety of emergency cases gives me the appropriate exposure

On this unit the types of operations performed are too complex for my level*

0.92

0.73

0.75

0.81

Optimal student-teacher interaction & opportunities for equal participation

I get enough opportunity to assist

There are enough theatre sessions per week for me to gain the appropriate experience

The anaesthetists put pressure on my trainer to operate himself to reduce anaesthetic time*

I am too busy doing other work to go to theatre*

When I am in theatre, there is nobody to cover the ward*

0.65

0.76

0.63

0.51

0.42

QUALITY OF FACULTY &TRAINING

(Educator Related Factors)

Interest of educator

My trainer is enthusiastic about teaching

"The first item in this section is quite representative, relevant, and clear in terms of assessing the interest of the teacher, which to many students, remains a central component."

"This section, in terms of the number of items remains under-represented."

“ mini-STEEM does not effectively evaluate the importance of teacher’s competence, teaching style, and educator’s preparedness.”

0.95

Importance of the educator's behavior and attitude

No representative Item

Competency of educator

No representative Item

Importance of teaching style

No representative Item

Importance of teacher’s preparedness

No representative Item

ORGANIZATIONAL SUPPORT

Significance of OR orientation session.

No representative Item

"The only item of this section is quite representative, but at best would only evaluate environmental readiness to a subjective extent. We need items to assess the provision and use of technology for better visualization of the procedures and synchronization with the simulation lab or simulated-OR based learning."

"Orientation session about the working within the theatre, sterilization protocols, patient safety measures, etc. all can be quite helpful."

Importance of environmental readiness

The atmosphere in theatre is pleasant

0.87

Synchronization simulation / Lab activities with OR lessons

No representative Item

Importance of adequate visualization in student learning

No representative Item

PSYCHOSOCIAL MANAGEMENT & TRAINING

Impact of anxiety in the OR environment

No representative Item

“mini-STEEM quite rightly assesses the discrimination based on race—quite valuable items”

"It also effectively judges the role of a friendly environment and its impact on student learning."

"It does not, however, assess any mechanism to address the issue of intimidation and victimization within the OR environment, the mechanism for redressal and role of faculty training in this regard."

Effect of fear, intimidation, discrimination or victimization in OR learning environment

I feel discriminated against in theatre because of my sex*

I feel discriminated against in theatre because of my race*

0.83

0.81

Impact of feeling welcome in OR

The theatre staff are friendly

0.88

STUDENT’S SELF-REGULATION, MOTIVATION & PARTICIPATION

Ability to self-regulate learning in OR

No representative Item

"According to Lyon's model of OR-based learning, student's capacity to self-regulate is of pivotal importance and is an independent predictor of student's quality of OR experience."

"The inventory does not assess student's talent to self-regulate, readiness to participate, and motivation to make this learning process more productive."

“Likewise, mini-STEEM does not explore the role of student's self-confidence, focus on practicing the motor and affective skills, and pre-operative self-review of the learning material to prime for a more enriched learning experience."

Impact of student’s motivation

No representative Item

Impact of self confidence

No representative Item

Student’s Prior Knowledge & Pre-lesson self-review of reading material

No representative Item

Student's readiness to participate

No representative Item

Student’s Focus on Practice of skills

No representative Item

Figure 1: Flow Chart describing the components of study and instrument evaluation

Figure 2: Factors affecting student learning in Operating Room: A conceptual Framework

In phase 2, the scale was content validated by 8 experts, and the I-CVI estimation was done as described previously. The qualitative data regarding the mini-STEEM items was subjected to thematic analysis.

The mini-STEEM scale was piloted among 202 students. The mean item scores and their respective SD has been described in Table 2. Exploratory factor analysis (Principal Component Analysis) was done to check for the construct validity, and Cronbach alpha estimation was done to evaluate the reliability of the constructs (Table 3).

Table 2: mini-STEEM Item Scores, Means and SD

Sr. No	Mini-STEEM Items	N	Min	Max	Mean	Std. Deviation
1	My trainer is enthusiastic about teaching.	202	1.0	5.0	2.327	1.1251
2	The theatre staff are friendly.	202	1.0	5.0	2.535	1.2303
3	There are enough theatre sessions per week for me to gain the appropriate experience.	202	1.0	5.0	2.866	1.3147
4	Before the operation my trainer discusses the surgical Technique planned.	202	1.0	5.0	2.782	1.1642
5	The elective operating list has the right case mix to suit my training.	200	1.0	5.0	2.770	1.0830
6	The variety of emergency cases gives me the appropriate exposure.	202	1.0	5.0	2.896	1.2634
7	I get enough opportunity to assist.	202	3.0	5.0	4.213	.7722
8	On this unit the types of operations performed are too complex for my level.*	202	3.0	5.0	3.851	.7037
9	The anaesthetists put pressure on my trainer to operate himself to reduce anaesthetic time*	202	3.0	5.0	3.955	.8364
10	I feel discriminated against in theatre because of my sex*	202	1.0	5.0	3.663	1.3985
11	I feel discriminated against in theatre because of my race*	202	1.0	5.0	4.069	1.3696
12	I am too busy doing other work to go to theatre*	202	3.0	5.0	4.158	.7562
13	The atmosphere in theatre is pleasant.	202	3.0	5.0	3.995	.7498
	Total				44.082

Table 3: Reliability Factor Loadings related to Principal Component Analysis of mini-STEEM

Rotated Component Matrix ^a						Cronbach's Alpha	Cronbach's Alpha Based on Standardized Items	N of Items
	Component
		1		2	3			4
6. The variety of emergency cases gives me the appropriate exposure.	.740		.224	-.073	.116	.737	.739	5
3. There are enough theatre sessions per week for me to gain the appropriate experience.	.713		-.044	.056	-.002
5. The elective operating list has the right case mix to suit my training.	.671		-.117	-.018	-.132
4. Before the operation my trainer discusses the surgical Technique planned.	.650		-.047	-.091	.048
1. My trainer is enthusiastic about teaching.	.593		-.562	-.099	-.094
11. I feel discriminated against in theatre because of my race.	.126		.844	-.055	.060	0.696	0.697	2
10. I feel discriminated against in theatre because of my sex.	-.005		.813	.218	-.121
2. The theatre staff are friendly.	.463		-.548	.061	.043	-
8. On this unit the types of operations performed are too complex for my level.	.091		.010	.780	.002	.537	.539	3
9. The anesthetists put pressure on my trainer to operate himself to reduce anesthetic time.	.039		.137	.683	.239
13. The atmosphere in theatre is pleasant.	-.341		-.020	.643	.010
7. I get enough opportunity to assist.	.183		-.105	.000	.777	.255	.255	2
12. I am too busy doing other work to go to theatre.	-.211		.063	.224	.683
Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization. a. Rotation converged in 5 iterations.		KMO and Bartlett's Test Kaiser-Meyer-Olkin Measure of Sampling Adequacy..753 Bartlett's Test of SphericityApprox. Chi-Square513.116 df78 Sig..000

The statistical analysis was done in SPSS, and SPSS Amos, and Cronbach alpha estimation, and the content validity index were measured as described previously.

Results

In the first phase, the literature evidence was explored, and five basic domains of factors affecting the students' learning experience within the OR were identified as tabulated in Table 1. The corresponding conceptual framework has been described in Figure 2. These domains include structured learning process, quality of faculty and training, psychosocial management, organizational support, and students' ability to self-regulate. Based on these 5 themes, 26 different sub-themes were identified, tabulated in Table 1.

In Phase 2 of this study, mini-STEEM items were content validated by 8 experts. The I-CVI and the corresponding expert analysis of these items have been summarized in Table 1, along with their representative expert quotes.

Based on literature evidence, the experts consider the structured learning process to be one of the most critical factors influencing the quality of overall learning experience within the OR setting. Mini-STEEM has 9 items to assess the structured learning process, concentrating mostly on the process of content selection and quality of student-teacher interaction. However, these items do not determine structured vs. opportunistic learning, the learning objectives' design and their adequate communication to the students.

Structured Learning Process:

The structured learning approach provides a framework for students to follow and includes specified preassigned readings, immediate feedback from their tutors following a clinical experience and the use of systematically organized instructional methodologies. This approach is also effective in enforcing the deliberate practice of clinical skills acquisition.

According to one expert, “The students prefer a structured and well-planned learning process within the operating theatre realm instead of an opportunistic learning process—this inventory does not adequately assess whether learning is structured or opportunistic."

For the OR experience to be beneficial, it is of paramount importance that practical learning objectives are formulated and are communicated with the students.The involvement of learners in the formation of learning objectives allows them to recognize the areas where skills need to be acquired. With regard to learning objectives, an expert pointed out that "like any learning process, learning objectives should be formulated prior to the lesson and following the convention of medical education, should be specific, measurable, attainable, relevant, targeted & time-bound (SMART)…these learning objectives should conform to the rest of teaching…… students should have room for their learning objectives to promote diversity in the learning process—Mini-STEEM does not address these important themes."

Moreover, another expert stated that "the items on the content selection are quite relevant; however, they do not represent all of the factors influencing this process. For example, it does not assess the process of curricular design and students' involvement in this whole process. It does assess the 'briefing' component of Robert's "Briefing-Intraoperative Teaching-Debriefing" model; however, it does not appraise the mechanism of intraoperative teaching and debriefing session." An intraoperative discussion can be developed from the set learning objectives in the briefing session. This model ensures that no operating room encounter goes purposeless for the learner and allows the preceptor and the learner to reflect on and reinforce the lessons learnt. Not addressing these factors renders the evaluation made by mini-STEEM inadequate.

Furthermore, an expert commented that "mini-STEEM overlooks and does not evaluate the process of assessment or delivery of constructive feedback to the student about their performance in OR learning."

And another stating that “the instrument does not gauge the process of assessment of skills and affective domains of learning.”

Surgical residents and medical students differ not only in their degree of knowledge and experience but also in their responsibilities and expectations from the OR encounter. It is therefore, impracticable to assess their needs and influencing factors on the same scale. As stated by an expert "in mini-STEEM, theatre sessions reflect a segment of opportunity for learning. The items, however, have been primarily designed for the residents, and it reflects in the structure of these items. For example, it may be quite hard to provide the opportunity to scrub along with the surgeon and make this a standard, considering the increasing number of students and residents. The last three items clearly pertain to residents, not medical students."

Quality Of Faculty & Training

Regarding the statement, “My trainer is enthusiastic about teaching” from mini-STEEM, one expert agreed that "the first item in this section is quite representative, relevant, and clear in terms of assessing the interest of the teacher, which for many students, remains a central component."

However, when the subtheme ‘competency of educator’ was questioned, an expert responded that "this section, in terms of the number of items remains under-represented."

Another expert observed that “mini-STEEM does not effectively evaluate the importance of teacher’s competence, teaching style, and educator’s preparedness.”

Organizational Support

The availability of basic and fundamental technology can have a remarkable impact on

the theater-based learning.

It was highlighted by an expert that the statement ‘the atmosphere in the theatre is pleasant’ is "the only item of this section that is quite representative, but at best would only evaluate environmental readiness to a subjective extent. We need items to assess the provision and use of technology for better visualization of the procedures and synchronization with the simulation lab or simulated-OR based learning."

It was further added that an "orientation session about working within the theatre, sterilization protocols, patient safety measures, etc. all can be quite helpful."

Psychosocial Management & Training

Medical students are subjected to a wide array of stimuli within the OR setting which can trigger a range of mixed emotions. Students must face new challenges whilst overcoming their emotions.

According to an expert "mini-STEEM quite rightly assesses the discrimination based on race—quite valuable items."

And in addition to that "it also effectively judges the role of a friendly environment and its impact on student learning."

Another expert further stated that "it does not, however, assess any mechanism to address the issue of intimidation and victimization within the OR environment, the mechanism for redressal and role of faculty training in this regard."

Student’s Self-Regulation, Motivation & Participation

Based on the literature review, it has been seen that students’ attitude and their level of interest and responsiveness determines the drive of the surgeon to teach.

It was stated by an expert that "according to Lyon's model of OR-based learning, a student's capacity to self-regulate is of pivotal importance and is an independent predictor of student's quality of OR experience."

Regarding mini-STEEM, an expert mentioned that "the inventory does not assess students' talent to self-regulate, readiness to participate, and motivation to make this learning process more productive."

And "likewise, mini-STEEM does not explore the role of students' self-confidence, focus on practicing the motor and affective skills, and pre-operative self-review of the learning material to prime for a more enriched learning experience."

Discussion

Medical students’ learning experience is unique, differs from a resident's learning perspective, and is multifactorial. Previously, experts in this area have identified and prioritized various intermediary items, which regulate the learning process and quality outcomes. Many factors related to the structured learning process, organization, educator, student, and psychosocial domains determine the quality of overall learning experience in the OR setting. Previously, mini-STEEM, an abbreviated version of STEEM, has been used to assess the OR learning process's quality. However, its content validity has been challenged recently, considering expanding dimensions of the OR learning process.

There is a growing body of literature that directs and indicates the expanding role of a structured learning process despite the difficulties of its implementation within the OR setting. The structured learning process involves careful course planning, delivery process, and assessment involving the psychomotor and affective components apart from the operative learning's cognitive portion. Similarly, educator-related and student-related factors significantly influence the learning process within the OR setting. Teacher's interest, competence, style, and welcoming attitude are independent predictors of quality OR learning. Faculty training in this regard can be pivotal and also provides chances for better student handling within the OR setting. The students' interest, receptive attitude, capacity to self-regulate learning and social handling within the OR can alter the outcomes of the learning process. The organizational support in terms of providing adequate technology and administrative support has additive value. Psychosocial training of the faculty and the students can positively affect the quality of learning process.

[1] Croghan, S., Phillips, C., Howson, W. (2019). The operating theatre as a classroom: a literature review of medical student learning in the theatre environment. International Journal of Medical Education https://doi.org/10.5116/ijme.5ca7.afd1

[2] Hexter, A., O’Dowd-Booth, C., Hunter, A. (2018). Factors that influence medical student learning in the operating room. Medical Teacher https://doi.org/10.1080/0142159X.2018.1504163

[3] Lyon, P. (2003). Making the most of learning in the operating theatre: Student strategies and curricular initiatives. Medical Education 37(8), 680-688 https://doi.org/10.1046/j.1365-2923.2003.01583.x

[4] Roberts, N., Brenner, M., Williams, R., Kim, M., Dunnington, G. (2012). Capturing the teachable moment: A grounded theory study of verbal teaching interactions in the operating room. Surgery 151(5), 643-650 https://doi.org/10.1016/j.surg.2011.12.011

[5] Nagraj, S., Wall, D., Jones, E. (2007). The development and validation of the mini-surgical theatre educational environment measure. Medical Teacher https://doi.org/10.1080/01421590701299249

[6] Cassar, K. (2004). Development of an instrument to measure the surgical operating theatre learning environment as perceived by basic surgical trainees. Medical Teacher 26(3), 260-264 https://doi.org/10.1080/0142159042000191975

[7] Waseem, T., Munir Baig, H., Yasmin, R., Ahmad Khan, R. (2020). Exploring Operating Room-Based Student Learning Experience: Perils & Pitfalls? A Narrative Literature Review.. Archives of Surgical Research 1(1), 38-46

[8] Roberts, N., Williams, R., Kim, M., Dunnington, G. (2009). The Briefing, Intraoperative Teaching, Debriefing Model for Teaching in the Operating Room. Journal of the American College of Surgeons 208(2), 299-303 https://doi.org/10.1016/j.jamcollsurg.2008.10.024

[9] Shetty, S., Zevin, B., Grantcharov, T., Roberts, K., Duffy, A. (2014). Perceptions, training experiences, and preferences of surgical residents toward laparoscopic simulation training: A resident survey. Journal of Surgical Education https://doi.org/10.1016/j.jsurg.2014.01.006

[10] Lyon, P. (2004). A model of teaching learning in the operating theatre. Medical Education https://doi.org/10.1111/j.1365-2929.2004.02020.x

[11] Weinberg, D., Saleh, M., Sinha, Y. (2015). Twelve tips for medical students to maximise learning in theatre. Medical Teacher https://doi.org/10.3109/0142159X.2014.932899

[12] Cope, A., Bezemer, J., Sutkin, G. (2019). . https://doi.org/10.1007/978-981-13-3128-2_16

Revalidating “Mini Surgical Theater Educational Environment Measure (mini-STEEM)”: Is it a Content Valid Instrument?

Authors

Keywords:

Abstract

Downloads

Published

How to Cite

Issue

Section

License

Revalidating “Mini Surgical Theater Educational Environment Measure (mini-STEEM)”: Is it a Content Valid Instrument?

Introduction

Methodology

Results

Structured Learning Process:

Quality Of Faculty & Training

Organizational Support

Psychosocial Management & Training

Student’s Self-Regulation, Motivation & Participation

Discussion


	Information for Authors
	Information for Readers
	Information for Librarians