Appraisal of Current Guidelines Regarding The Management Of Breast Cancer Using the Appraisal Of Guidelines Research and Evaluation (AGREE) Instrument: A Critical Review

Authors

  • Hadia Baig
  • Talat Waseem Shalamar Medical & Dental College

DOI:

https://doi.org/10.48111/2021.03.03

Keywords:

breast cancer, management, AGREE II Instrument, Clinical practice guidelines, guidelines, consensus, quality of guidelines

Abstract

With rapid advancements in breast oncology, there is a growing need for high-quality, systematically developed clinical practice guidelines (CPGs) and consensus statements (CSs). This study aims to assess the quality of the current clinical practice guidelines and consensus statements related to the management of breast cancer employing the Appraisal of Guidelines Research and Evaluation (AGREE) II tool.

METHODS A systematic literature search of bibliographic databases (PubMed and Google Scholar) and 20 professional society websites was conducted from January 2019 onwards. Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument was used to evaluate the methodological quality of the included CPGs and CSs.

 RESULTS The analysis of the AGREE II overall assessment of CPGs and CSs revealed a wide overall score range. The median overall score across the guidelines was 61%. The highest overall score was obtained by the American Society of Clinical Oncology (ASCO) guidelines, with scores ranging from 80-91%, followed by 5th ESO-ESMO ABC5 (77%), 4th ESO-ESMO BCY4 (76%) and V.4 NCCN (74%). Overall, CSs had a lower quality in the majority of the domains as compared to CPGs.

 CONCLUSIONS The authors believe that the guidelines related to breast cancer management have a wide room for improvement. There is a growing need for CPGs/CSs that employ uniformly endorsed standards. Guideline development standards are the current state-of-the-art, and guideline developers must direct their efforts towards acknowledging and incorporating them into guidelines.

Author Biography

Talat Waseem, Shalamar Medical & Dental College

Dr. Talat Waseem, FRCS Eng., FACS, DM (Harvard), Associate Professor in the Department of Surgery, Shalamar Medical & Dental College, Lahore, Pakistan

References

Organization WH. Breast Cancer. Published 2021. https://www.who.int/news-room/fact-sheets/detail/breast-cancer

Lei X, Liu F, Luo S, et al. Evaluation of guidelines regarding surgical treatment of breast cancer using the AGREE Instrument: a systematic review. BMJ open. 2017;7(11):e014883. doi:10.1136/bmjopen-2016-014883

Kumar Tyagi N, Dhesy-Thind S. Clinical practice guidelines in breast cancer. Current Oncology. 2018;25:S151-S160. doi:10.3747/CO.25.3729

Maes-Carballo M, Mignini L, Martín-Díaz M, Bueno-Cavanillas A, Khan KS. Quality and reporting of clinical guidelines for breast cancer treatment: A systematic review. Breast (Edinburgh, Scotland). 2020;53:201-211. doi:10.1016/j.breast.2020.07.011

Brouwers MC, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, Fervers B, Graham ID, Grimshaw J H, SE, Littlejohns P, Makarski J ZL. AGREE Next Steps Consortium. AGREE II: Advancing guideline development, reporting and evaluation in healthcare. CMAJ. Published online 2010:182:E839-842. https://www.agreetrust.org/wp-content/uploads/2017/12/AGREE-II-Users-Manual-and-23-item-Instrument-2009-Update-2017.pdf

D M, A L, J T, DG A. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS medicine. 2009;6(7). doi:10.1371/JOURNAL.PMED.1000097

Vernooij RWM, Sanabria AJ, Solà I, Alonso-Coello P, Martínez García L. Guidance for updating clinical practice guidelines: a systematic review of methodological handbooks. Implementation science : IS. 2014;9:3. doi:10.1186/1748-5908-9-3

Hoffmann-Eßer W, Siering U, Neugebauer EAM, Lampert U, Eikermann M. Systematic review of current guideline appraisals performed with the Appraisal of Guidelines for Research & Evaluation II instrument—a third of AGREE II users apply a cut-off for guideline quality. Journal of Clinical Epidemiology. 2018;95:120-127. doi:10.1016/J.JCLINEPI.2017.12.009

Cancer Australia, Surry Hills NSW. Guidance for the management of early breast cancer: Recommendations and practice points. Published online 2020. https://www.canceraustralia.gov.au/publications-and-resources/cancer-australia-publications/guidance-management-early-breast-cancer-recommendations-and-practice-points

Ditsch N, Kolberg-Liedtke C, Friedrich M, et al. AGO Recommendations for the Diagnosis and Treatment of Patients with Early Breast Cancer: Update 2021. Breast care (Basel, Switzerland). 2021;16(3):214-227. doi:10.1159/000516419

Thill M, Jackisch C, Janni W, et al. AGO Recommendations for the Diagnosis and Treatment of Patients with Locally Advanced and Metastatic Breast Cancer: Update 2019. Breast care (Basel, Switzerland). 2019;14(4):247-255. doi:10.1159/000500999

Unit GR. Systemic Therapy for Early Breast Cancer - Quick Reference Guide -. 2021;(April). https://www.albertahealthservices.ca/assets/info/hp/cancer/if-hp-cancer-guide-systemic-therapy-early-breast.pdf

Hassett MJ, Somerfield MR, Baker ER, et al. Management of Male Breast Cancer: ASCO Guideline. Journal of clinical oncology : official journal of the American Society of Clinical Oncology. 2020;38(16):1849-1863. doi:10.1200/JCO.19.03120

Korde LA, Somerfield MR, Carey LA, et al. Neoadjuvant Chemotherapy, Endocrine Therapy, and Targeted Therapy for Breast Cancer: ASCO Guideline. Journal of clinical oncology : official journal of the American Society of Clinical Oncology. 2021;39(13):1485-1505. doi:10.1200/JCO.20.03399

Visvanathan K, Fabian CJ, Bantug E, et al. Use of Endocrine Therapy for Breast Cancer Risk Reduction: ASCO Clinical Practice Guideline Update. Journal of clinical oncology : official journal of the American Society of Clinical Oncology. 2019;37(33):3152-3165. doi:10.1200/JCO.19.01472

Denduluri N, Somerfield MR, Chavez-MacGregor M, et al. Selection of Optimal Adjuvant Chemotherapy and Targeted Therapy for Early Breast Cancer: ASCO Guideline Update. Journal of clinical oncology : official journal of the American Society of Clinical Oncology. 2021;39(6):685-693. doi:10.1200/JCO.20.02510

Brackstone M, Baldassarre FG, Perera FE, et al. Management of the Axilla in Early-Stage Breast Cancer: Ontario Health (Cancer Care Ontario) and ASCO Guideline. Journal of clinical oncology : official journal of the American Society of Clinical Oncology. 2021;39(27):3056-3082. doi:10.1200/JCO.21.00934

Burstein HJ, Somerfield MR, Barton DL, et al. Endocrine Treatment and Targeted Therapy for Hormone Receptor-Positive, Human Epidermal Growth Factor Receptor 2-Negative Metastatic Breast Cancer: ASCO Guideline Update. Journal of clinical oncology : official journal of the American Society of Clinical Oncology. Published online July 29, 2021:JCO2101392. doi:10.1200/JCO.21.01392

Tung NM, Zakalik D, Somerfield MR, Hereditary Breast Cancer Guideline Expert Panel. Adjuvant PARP Inhibitors in Patients With High-Risk Early-Stage HER2-Negative Breast Cancer and Germline BRCA Mutations: ASCO Hereditary Breast Cancer Guideline Rapid Recommendation Update. Journal of clinical oncology : official journal of the American Society of Clinical Oncology. 2021;39(26):2959-2961. doi:10.1200/JCO.21.01532

Moy B, Rumble RB, Come SE, et al. Chemotherapy and Targeted Therapy for Patients With Human Epidermal Growth Factor Receptor 2-Negative Metastatic Breast Cancer That is Either Endocrine-Pretreated or Hormone Receptor-Negative: ASCO Guideline Update. Journal of clinical oncology : official journal of the American Society of Clinical Oncology. Published online July 29, 2021:JCO2101374. doi:10.1200/JCO.21.01374

Manoj Gowda S, Kabeer KK, Jafferbhoy S, et al. Breast Cancer Management Guidelines During COVID-19 Pandemic. The Indian journal of surgery. Published online July 2, 2020:1-8. doi:10.1007/s12262-020-02466-7

Jiao D-C, Zhu J-J, Qin L, et al. Clinical practice guidelines for modified radical mastectomy of breast cancer: Chinese Society of Breast Surgery (CSBrs) practice guidelines 2021. Chinese medical journal. 2021;134(8):895-897. doi:10.1097/CM9.0000000000001412

Liu M, Wang C-B, Xie F, Peng Y, Wang S, Chinese Society of Breast Surgery. Clinical practice guidelines for diagnosis and treatment of invasive breast cancer: Chinese Society of Breast Surgery (CSBrS) practice guidelines 2021. Chinese medical journal. 2021;134(9):1009-1013. doi:10.1097/CM9.0000000000001498

Ma L, Ge Z-C, Xin L, et al. Clinical practice guidelines for breast cancer implantable intravenous infusion ports: Chinese Society of Breast Surgery practice guidelines 2021. Chinese medical journal. 2021;134(16):1894-1896. doi:10.1097/CM9.0000000000001522

Chen K, Liu J-Q, Wu W, et al. Clinical practice guideline for breast-conserving surgery in patients with early-stage breast cancer: Chinese Society of Breast Surgery (CSBrS) practice guidelines 2021. Chinese medical journal. Published online June 16, 2021. doi:10.1097/CM9.0000000000001518

Cardoso F, Kyriakides S, Ohno S, et al. Early breast cancer: ESMO Clinical Practice Guidelines for diagnosis, treatment and follow-up†. Annals of oncology : official journal of the European Society for Medical Oncology. 2019;30(8):1194-1220. doi:10.1093/annonc/mdz173

Martin M, Guerrero-Zotano A, Montero Á, et al. GEICAM Guidelines for the Management of Patients with Breast Cancer During the COVID-19 Pandemic in Spain. The oncologist. 2020;25(9):e1339-e1345. doi:10.1634/theoncologist.2020-0363

Daraï E, Mathelin C, Gligorov J. Breast Cancer Management during the COVID 19 Pandemic: French Guidelines. European journal of breast health. 2020;16(3):160-161. doi:10.5152/ejbh.2020.200420

Inokuchi M, Kutomi G, Kijima Y, et al. The Japanese Breast Cancer Society clinical practice guidelines for surgical treatment of breast cancer, 2018 edition. Breast cancer (Tokyo, Japan). 2020;27(1):4-8. doi:10.1007/s12282-019-01030-w

Shimoi T, Nagai SE, Yoshinami T, et al. The Japanese Breast Cancer Society Clinical Practice Guidelines for systemic treatment of breast cancer, 2018 edition. Breast cancer (Tokyo, Japan). 2020;27(3):322-331. doi:10.1007/s12282-020-01085-0

Iwata H, Saji S, Ikeda M, et al. The Japanese Breast Cancer Society Clinical Practice Guidelines, 2018 edition: the tool for shared decision making between doctor and patient. Breast cancer (Tokyo, Japan). 2020;27(1):1-3. doi:10.1007/s12282-019-01021-x

National Health Commission Of The People’s Republic Of China. Chinese guidelines for diagnosis and treatment of breast cancer 2018 (English version). Chinese journal of cancer research = Chung-kuo yen cheng yen chiu. 2019;31(2):259-277. doi:10.21147/j.issn.1000-9604.2019.02.02

Mcneill C. Breast Cancer Clinical Guidelines Title: NCA Breast Cancer Clinical Guidelines Authors: Breast EAG members Circulation List: Breast EAG. 2020;(May 2016):1-117. https://www.northerncanceralliance.nhs.uk/wp-content/uploads/2019/02/NCA-Breast-Cancer-Guidelines-v2-.10.pdf

Gradishar WJ, Moran MS, Abraham J, et al. NCCN Guidelines® Insights: Breast Cancer, Version 4.2021. Journal of the National Comprehensive Cancer Network : JNCCN. 2021;19(5):484-493. doi:10.6004/jnccn.2021.0023

Ribociclib with fulvestrant for treating hormone receptor-positive, HER2-negative, advanced breast cancer | Guidance | NICE 1 Recommendations |. 2021;(March). https://www.nice.org.uk/guidance/ta593/chapter/1-Recommendations

NICE. Trastuzumab emtansine for adjuvant treatment of HER2-positive early breast cancer 1 Recommendations. 2018;(June).

NICE. Trastuzumab deruxtecan for treating HER2-positive unresectable or metastatic breast cancer after 2 or more anti-HER2 therapies. 2021;(May). www.nice.org.uk/guidance/ta704

NICE TA579. Abemaciclib with fulvestrant for treating hormone HER2-negative advanced breast cancer after endocrine therapy. 2020;(May 2019).

National institute for Health and Care Excellence N. Pertuzumab for adjuvant treatment of e early stage breast cancer. 2019;(March 2019):16.

National Institute for Health and Care Excellence (NICE). Atezolizumab with nab-paclitaxel for untreated PD-L1-positive, locally advanced or metastatic, triple-negative breast cancer. Technology appraisal guidance [TA639]. 2020;(July):1-19.

Khan S, Richardson J. Palbociclib with fulvestrant for treating hormone breast cancer. 2020;(January):20.

National Institute for Health and Care Excellence (NICE). Neratinib for extended adjuvant treatment of hormone receptor-positive, HER2-positive early stage breast cancer after adjuvant trastuzumab. Technology appraisal guidance [TA612]. 2019;(November 2019). https://www.nice.org.uk/guidance/TA612

National Institute for Health and Care Excellence. Abemaciclib with an aromatase inhibitor for previously untreated, hormone receptor-positive, HER2-negative, locally advanced or metastatic breast cancer. Technology appraisal guidance [TA563]. 2019;(February 2019). https://www.nice.org.uk/guidance/ta563

Dowsett M, Ellis MJ, Dixon JM, et al. Evidence-based guidelines for managing patients with primary ER+ HER2? breast cancer deferred from surgery due to the COVID-19 pandemic. npj Breast Cancer. 2020;6(1):21. doi:10.1038/s41523-020-0168-9

Simpson LK, Chin SN, East J, et al. Guidelines on management of the patient with breast cancer. West Indian Medical Journal. 2019;68:55-72. doi:10.7727/wimj.2018.198

American Society of Breast Surgeons T. Consensus Guideline on the Management of the Axilla in Patients With Invasive/In-Situ Breast Cancer Purpose. Published online 2014:1-12. https://www.breastsurgeons.org/docs/statements/Consensus-Guideline-on-the-Management-of-the-Axilla.pdf?v2

Chacón López-Muñiz JI, de la Cruz Merino L, Gavilá Gregori J, et al. SEOM clinical guidelines in advanced and recurrent breast cancer (2018). Clinical & translational oncology : official publication of the Federation of Spanish Oncology Societies and of the National Cancer Institute of Mexico. 2019;21(1):31-45. doi:10.1007/s12094-018-02010-w

Cardoso F, Paluch-Shimon S, Senkus E, et al. 5th ESO-ESMO international consensus guidelines for advanced breast cancer (ABC 5). Annals of oncology : official journal of the European Society for Medical Oncology. 2020;31(12):1623-1649. doi:10.1016/j.annonc.2020.09.010

Xu B, Hu X, Feng J, et al. Chinese expert consensus on the clinical diagnosis and treatment of advanced breast cancer (2018). Cancer. 2020;126 Suppl:3867-3882. doi:10.1002/cncr.32832

Paluch-Shimon S, Cardoso F, Partridge AH, et al. ESO-ESMO 4th International Consensus Guidelines for Breast Cancer in Young Women (BCY4). Annals of oncology : official journal of the European Society for Medical Oncology. 2020;31(6):674-696. doi:10.1016/j.annonc.2020.03.284

Burstein HJ, Curigliano G, Thürlimann B, et al. Customizing local and systemic therapies for women with early breast cancer: the St. Gallen International Consensus Guidelines for treatment of early breast cancer 2021. Annals of oncology : official journal of the European Society for Medical Oncology. 2021;32(10):1216-1235. doi:10.1016/j.annonc.2021.06.023

Nava MB, Benson JR, Audretsch W, et al. International multidisciplinary expert panel consensus on breast reconstruction and radiotherapy. The British journal of surgery. 2019;106(10):1327-1340. doi:10.1002/bjs.11256

Bešlija S, Gojkovi? Z, Ceri? T, et al. 2020 consensus guideline for optimal approach to the diagnosis and treatment of HER2-positive breast cancer in Bosnia and Herzegovina. Bosnian journal of basic medical sciences. 2021;21(2):120-135. doi:10.17305/bjbms.2020.4846

Downloads

Published

2021-09-29

How to Cite

Baig, H., & Waseem, T. (2021). Appraisal of Current Guidelines Regarding The Management Of Breast Cancer Using the Appraisal Of Guidelines Research and Evaluation (AGREE) Instrument: A Critical Review. Archives of Surgical Research, 2(3), 16-26. https://doi.org/10.48111/2021.03.03

Issue

Vol. 2 No. 3 (2021): July-September

Section

Original Research Communication

License

Copyright (c) 2021 Hadia Baig, Talat Waseem

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Introduction

Operating Room (OR) based learning experience has traditionally remained challenging owing to complex medical, psychosocial, educational, and administrative factors, and so has its measurement. Recently, mini-STEEM, an abbreviated form of Surgical Theater Educational Experience Measure (STEEM), has been employed to evaluate medical students’ OR-based learning experience. This study aims to assess its content and construct validity in light of an updated extensive literature review.

Methods

Extensive literature review was done to identify and evaluate various factors affecting students' OR-based learning within the OR-setting. Expert validation of the items was done in light of constructs identified through literature review by 8 experts who had experience both in surgical teaching and medical education. The item content validity index (I-CVI) was assessed. Thematic analysis of the qualitative responses was done to understand the dynamics of validity scores. Factor analysis was performed to see the factor loadings for the items and their model fitness.

Results

Against previously identified 26 constructs, mini-STEEM items were expert-validated for clarity, relevance, and comprehension. The item content validity index was suboptimal and did not cover all of the dimensions and constructs identified through a previous literature review. Reliability was checked through Cronbach’s alpha (0.779) after piloting through a cohort of students (n=156). Qualitative analysis revealed that mini-STEEM does not cover all of the domains essential for student learning within the OR setting, and does not have adequate representative items to address all elements. Exploratory factor analysis (EFA) showed reasonable loadings; however, the model fitness through confirmatory factor analysis (CFA) was not possible considering low or no representation of most of the themes.

Conclusion

In light of evolving literature evidence, the content validity of mini-STEEM to assess the quality of medical students' OR-based learning experience remains low, despite having good reliability and construct loadings.

Keywords

Operating Room; Operation Theater; Learning; Student; Resident; Quality of Learning Experience; Structured Learning; STEEM; mini-STEEM; Content validity

Introduction

The dynamic environment within the Operating Room (OR) provides a real-life venule for medical students’ learning1 2.However, there are many medical, social, ethical, and administrative factors involved, which make learning within the OR setting exigent for teachers and learners both2 3. Recently, many experts in this area have generated a new conceptual framework highlighting various important factors that influence the quality of OR-based student learning1 2.Lyon's model focuses on student-driven learning within the OR, while Roberts et al. stress a more deliberate learning process for the medical students 3 4.There is a consensus that the learning process within at large remains opportunistic, unstructured, un-driven, and hence substandard 1 2.Consequently, the OR setting's standardized learning process and quality assurance remain the key for enhanced OR learning.

Nagraj et al. previously proposed the instrument ‘mini-STEEM’ to assess the quality of student learning within the OR environment, which is an abbreviated form of Kevin Cassar’s Surgical Theater Educational Environment Measure5 6 .This instrument by Kevin Cassar was primarily designed for the residents and is a reliable and content valid6. Recently in multiple systematic literature reviews, many important OR-based learning determinants have been identified, and a more comprehensive conceptual framework has been developed 1 2 7. Mini-STEEM, although reliable with reasonable Cronbach’s alpha value, does not assess all domains which pertain to medical students' learning experience within the OR setting.12

In light of an improved understanding of OR-based learning and multiple recent literature reviews that identify many essential domains of student learning within the OR setting, there is a clear need for re-validation of the mini-STEEM. This study aims to assess the content validity, reliability, and construct validity of mini-STEEM.

Methodology

Following ethical approval from the local institutional review, a board study was conducted, as shown in Figure 1. In the first phase, we relied on the previously published literature evidence8 8 9 .The literature search was conducted on PubMed, ERIC, and Google Scholar. Thematic analysis and review were performed to identify the factors influencing OR-based student learning. Additionally, the study consisted of the analysis of various models and scales currently being used to assess the OR-based student learning8 10 11.The conceptual framework and previously identified themes were used as reference (Table 1 and Figure 2).

“Table 1: Expert Validation of mini-STEEM based on Conceptual Framework”

Themes (5)

Subthemes (26)

Mini-STEEM Items

Representative Qualitative Statements

I-CVI

STRUCTURED LEARNING PROCESS

(Structured Course /Lesson Planning, Content Selection, Delivery, Assessment & Administration)

Structured or Opportunistic OR learning

No representative Item

“The students prefer a structured and well-planned learning process within operating theatre realm instead of opportunistic learning process—this inventory does not adequately assess whether learning is structured or opportunistic”

“Like any learning process learning objectives should be formulated prior to lesson and following the convention of medical education should be specific, measurable, attainable, relevant, targeted & time-bound (SMART)…these learning objectives should conform to the rest of teaching…… students should have room for their own personal learning objectives to promote diversity in learning process—Mini-STEEM does not address these important themes”

“The items pertaining to content selection are quite relevant, however they do not represent all of factors influencing this process. For example, it does not assess the process of curricular design and the involvement of students in this whole process. It does assess the ‘briefing’ component of Robert’s “Briefing-Intraoperative Teaching-Debriefing” model however does not appraise the mechanism of intraoperative teaching and debriefing session.”

“Mini-STEEM overlooks and does not evaluate the process of assessment or delivery of constructive feedback to the student about their performance in OR learning”

“The instrument does not gauge the process of assessment of skills and affective domains of learning.”

“In mini-STEEM, theatre sessions reflect a segment of opportunity for learning. The items however have been primarily designed for the residents and it reflects in structure of these items. For example, it may be quite hard to provide opportunity to scrub along with the surgeon and make this a standard considering the increasing number of students and residents. The last three items clearly pertain to residents not medical students.”

0

Design and Communication of Learning Objectives for OR learning

No representative Item

0

Clarity of Learning Objectives

No representative Item

0

Feasibility of learning objectives to be realistically achievable

No representative Item

0

Synchronization of the learning objectives with rest of the teaching.

No representative Item

0

Importance of Personal Learning Objectives in OR Learning

No representative Item

0

Content Selection, delivery, assessment and their alignment to essential aspects of graduate learning

Before the operation my trainer discusses the surgical technique planned

The elective operating list has the right case mix to suit my training

The variety of emergency cases gives me the appropriate exposure

On this unit the types of operations performed are too complex for my level*

0.92

0.73

0.75

0.81

Optimal student-teacher interaction & opportunities for equal participation

I get enough opportunity to assist

There are enough theatre sessions per week for me to gain the appropriate experience

The anaesthetists put pressure on my trainer to operate himself to reduce anaesthetic time*

I am too busy doing other work to go to theatre*

When I am in theatre, there is nobody to cover the ward*

0.65

0.76

0.63

0.51

0.42

QUALITY OF FACULTY &TRAINING

(Educator Related Factors)

Interest of educator

My trainer is enthusiastic about teaching

"The first item in this section is quite representative, relevant, and clear in terms of assessing the interest of the teacher, which to many students, remains a central component."

"This section, in terms of the number of items remains under-represented."

“ mini-STEEM does not effectively evaluate the importance of teacher’s competence, teaching style, and educator’s preparedness.”

0.95

Importance of the educator's behavior and attitude

No representative Item

0

Competency of educator

No representative Item

0

Importance of teaching style

No representative Item

0

Importance of teacher’s preparedness

No representative Item

0

ORGANIZATIONAL SUPPORT

Significance of OR orientation session.

No representative Item

"The only item of this section is quite representative, but at best would only evaluate environmental readiness to a subjective extent. We need items to assess the provision and use of technology for better visualization of the procedures and synchronization with the simulation lab or simulated-OR based learning."

"Orientation session about the working within the theatre, sterilization protocols, patient safety measures, etc. all can be quite helpful."

0

Importance of environmental readiness

The atmosphere in theatre is pleasant

0.87

Synchronization simulation / Lab activities with OR lessons

No representative Item

0

Importance of adequate visualization in student learning

No representative Item

0

PSYCHOSOCIAL MANAGEMENT & TRAINING

Impact of anxiety in the OR environment

No representative Item

“mini-STEEM quite rightly assesses the discrimination based on race—quite valuable items”

"It also effectively judges the role of a friendly environment and its impact on student learning."

"It does not, however, assess any mechanism to address the issue of intimidation and victimization within the OR environment, the mechanism for redressal and role of faculty training in this regard."

0

Effect of fear, intimidation, discrimination or victimization in OR learning environment

I feel discriminated against in theatre because of my sex*

I feel discriminated against in theatre because of my race*

0.83

0.81

Impact of feeling welcome in OR

The theatre staff are friendly

0.88

STUDENT’S SELF-REGULATION, MOTIVATION & PARTICIPATION

Ability to self-regulate learning in OR

No representative Item

"According to Lyon's model of OR-based learning, student's capacity to self-regulate is of pivotal importance and is an independent predictor of student's quality of OR experience."

"The inventory does not assess student's talent to self-regulate, readiness to participate, and motivation to make this learning process more productive."

“Likewise, mini-STEEM does not explore the role of student's self-confidence, focus on practicing the motor and affective skills, and pre-operative self-review of the learning material to prime for a more enriched learning experience."

0

Impact of student’s motivation

No representative Item

0

Impact of self confidence

No representative Item

0

Student’s Prior Knowledge & Pre-lesson self-review of reading material

No representative Item

0

Student's readiness to participate

No representative Item

0

Student’s Focus on Practice of skills

No representative Item

0

Figure 1: Flow Chart describing the components of study and instrument evaluation

Figure 2: Factors affecting student learning in Operating Room: A conceptual Framework

In phase 2, the scale was content validated by 8 experts, and the I-CVI estimation was done as described previously. The qualitative data regarding the mini-STEEM items was subjected to thematic analysis.

The mini-STEEM scale was piloted among 202 students. The mean item scores and their respective SD has been described in Table 2. Exploratory factor analysis (Principal Component Analysis) was done to check for the construct validity, and Cronbach alpha estimation was done to evaluate the reliability of the constructs (Table 3).

“Table 2: mini-STEEM Item Scores, Means and SD”

Sr. No

Mini-STEEM Items

N

Min

Max

Mean

Std. Deviation

1

My trainer is enthusiastic about teaching.

202

1.0

5.0

2.327

1.1251

2

The theatre staff are friendly.

202

1.0

5.0

2.535

1.2303

3

There are enough theatre sessions per week for me to gain the appropriate experience.

202

1.0

5.0

2.866

1.3147

4

Before the operation my trainer discusses the surgical Technique planned.

202

1.0

5.0

2.782

1.1642

5

The elective operating list has the right case mix to suit my training.

200

1.0

5.0

2.770

1.0830

6

The variety of emergency cases gives me the appropriate exposure.

202

1.0

5.0

2.896

1.2634

7

I get enough opportunity to assist.

202

3.0

5.0

4.213

.7722

8

On this unit the types of operations performed are too complex for my level.*

202

3.0

5.0

3.851

.7037

9

The anaesthetists put pressure on my trainer to operate himself to reduce anaesthetic time*

202

3.0

5.0

3.955

.8364

10

I feel discriminated against in theatre because of my sex*

202

1.0

5.0

3.663

1.3985

11

I feel discriminated against in theatre because of my race*

202

1.0

5.0

4.069

1.3696

12

I am too busy doing other work to go to theatre*

202

3.0

5.0

4.158

.7562

13

The atmosphere in theatre is pleasant.

202

3.0

5.0

3.995

.7498

Total

 

 

44.082

 

“Table 3: Reliability Factor Loadings related toPrincipal Component Analysis of mini-STEEM”

Rotated Component Matrix a

Cronbach's Alpha

Cronbach's Alpha Based on Standardized Items

N of Items

 

Component

1

2

3

4

6. The variety of emergency cases gives me the appropriate exposure.

.740

.224

-.073

.116

.737

.739

5

3. There are enough theatre sessions per week for me to gain the appropriate experience.

.713

-.044

.056

-.002

5. The elective operating list has the right case mix to suit my training.

.671

-.117

-.018

-.132

4. Before the operation my trainer discusses the surgical Technique planned.

.650

-.047

-.091

.048

1. My trainer is enthusiastic about teaching.

.593

-.562

-.099

-.094

11. I feel discriminated against in theatre because of my race.

.126

.844

-.055

.060

0.696

0.697

2

10. I feel discriminated against in theatre because of my sex.

-.005

.813

.218

-.121

2. The theatre staff are friendly.

.463

-.548

.061

.043

-

8. On this unit the types of operations performed are too complex for my level.

.091

.010

.780

.002

.537

.539

3

9. The anesthetists put pressure on my trainer to operate himself to reduce anesthetic time.

.039

.137

.683

.239

13. The atmosphere in theatre is pleasant.

-.341

-.020

.643

.010

7. I get enough opportunity to assist.

.183

-.105

.000

.777

.255

.255

2

12. I am too busy doing other work to go to theatre.

-.211

.063

.224

.683

Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization.

a. Rotation converged in 5 iterations.

KMO and Bartlett's Test

Kaiser-Meyer-Olkin Measure of Sampling Adequacy..753

Bartlett's Test of SphericityApprox. Chi-Square513.116

df78

Sig..000

The statistical analysis was done in SPSS, and SPSS Amos, and Cronbach alpha estimation, and the content validity index were measured as described previously[11].

Results

In the first phase, the literature evidence was explored, and five basic domains of factors affecting the students' learning experience within the OR were identified as tabulated in Table 1. The corresponding conceptual framework has been described in Figure 2. These domains include structured learning process, quality of faculty and training, psychosocial management, organizational support, and students' ability to self-regulate. Based on these 5 themes, 26 different sub-themes were identified, tabulated in Table 1.

In Phase 2 of this study, mini-STEEM items were content validated by 8 experts. The I-CVI and the corresponding expert analysis of these items have been summarized in Table 1, along with their representative expert quotes.

Based on literature evidence, the experts consider the structured learning process to be one of the most critical factors influencing the quality of overall learning experience within the OR setting. Mini-STEEM has 9 items to assess the structured learning process, concentrating mostly on the process of content selection and quality of student-teacher interaction. However, these items do not determine structured vs. opportunistic learning, the learning objectives' design and their adequate communication to the students.

STRUCTURED LEARNING PROCESS:

The structured learning approach provides a framework for students to follow and includes specified preassigned readings, immediate feedback from their tutors following a clinical experience and the use of systematically organized instructional methodologies. This approach is also effective in enforcing the deliberate practice of clinical skills acquisition.

According to one expert, “The students prefer a structured and well-planned learning process within the operating theatre realm instead of an opportunistic learning process—this inventory does not adequately assess whether learning is structured or opportunistic."

For the OR experience to be beneficial, it is of paramount importance that practical learning objectives are formulated and are communicated with the students.The involvement of learners in the formation of learning objectives allows them to recognize the areas where skills need to be acquired. With regard to learning objectives, an expert pointed out that "like any learning process, learning objectives should be formulated prior to the lesson and following the convention of medical education, should be specific, measurable, attainable, relevant, targeted & time-bound (SMART)…these learning objectives should conform to the rest of teaching…… students should have room for their learning objectives to promote diversity in the learning process—Mini-STEEM does not address these important themes."

Moreover, another expert stated that "the items on the content selection are quite relevant; however, they do not represent all of the factors influencing this process. For example, it does not assess the process of curricular design and students' involvement in this whole process. It does assess the 'briefing' component of Robert's "Briefing-Intraoperative Teaching-Debriefing" model; however, it does not appraise the mechanism of intraoperative teaching and debriefing session." An intraoperative discussion can be developed from the set learning objectives in the briefing session. This model ensures that no operating room encounter goes purposeless for the learner and allows the preceptor and the learner to reflect on and reinforce the lessons learnt. Not addressing these factors renders the evaluation made by mini-STEEM inadequate.

Furthermore, an expert commented that "mini-STEEM overlooks and does not evaluate the process of assessment or delivery of constructive feedback to the student about their performance in OR learning."

And another stating that “the instrument does not gauge the process of assessment of skills and affective domains of learning.”

Surgical residents and medical students differ not only in their degree of knowledge and experience but also in their responsibilities and expectations from the OR encounter. It is therefore, impracticable to assess their needs and influencing factors on the same scale. As stated by an expert "in mini-STEEM, theatre sessions reflect a segment of opportunity for learning. The items, however, have been primarily designed for the residents, and it reflects in the structure of these items. For example, it may be quite hard to provide the opportunity to scrub along with the surgeon and make this a standard, considering the increasing number of students and residents. The last three items clearly pertain to residents, not medical students."

QUALITY OF FACULTY & TRAINING

Regarding the statement, “My trainer is enthusiastic about teaching” from mini-STEEM, one expert agreed that "the first item in this section is quite representative, relevant, and clear in terms of assessing the interest of the teacher, which for many students, remains a central component."

However, when the subtheme ‘competency of educator’ was questioned, an expert responded that "this section, in terms of the number of items remains under-represented."

Another expert observed that “mini-STEEM does not effectively evaluate the importance of teacher’s competence, teaching style, and educator’s preparedness.”

ORGANIZATIONAL SUPPORT

The availability of basic and fundamental technology can have a remarkable impact on

the theater-based learning.

It was highlighted by an expert that the statement ‘the atmosphere in the theatre is pleasant’ is "the only item of this section that is quite representative, but at best would only evaluate environmental readiness to a subjective extent. We need items to assess the provision and use of technology for better visualization of the procedures and synchronization with the simulation lab or simulated-OR based learning."

It was further added that an "orientation session about working within the theatre, sterilization protocols, patient safety measures, etc. all can be quite helpful."

PSYCHOSOCIAL MANAGEMENT & TRAINING

Medical students are subjected to a wide array of stimuli within the OR setting which can trigger a range of mixed emotions. Students must face new challenges whilst overcoming their emotions.

According to an expert "mini-STEEM quite rightly assesses the discrimination based on race—quite valuable items."

And in addition to that "it also effectively judges the role of a friendly environment and its impact on student learning."

Another expert further stated that "it does not, however, assess any mechanism to address the issue of intimidation and victimization within the OR environment, the mechanism for redressal and role of faculty training in this regard."

STUDENT’S SELF-REGULATION, MOTIVATION & PARTICIPATION

Based on the literature review, it has been seen that students’ attitude and their level of interest and responsiveness determines the drive of the surgeon to teach.

It was stated by an expert that "according to Lyon's model of OR-based learning, a student's capacity to self-regulate is of pivotal importance and is an independent predictor of student's quality of OR experience."

Regarding mini-STEEM, an expert mentioned that "the inventory does not assess students' talent to self-regulate, readiness to participate, and motivation to make this learning process more productive."

And "likewise, mini-STEEM does not explore the role of students' self-confidence, focus on practicing the motor and affective skills, and pre-operative self-review of the learning material to prime for a more enriched learning experience."

Discussion

Medical students’ learning experience is unique, differs from a resident's learning perspective, and is multifactorial. Previously, experts in this area have identified and prioritized various intermediary items, which regulate the learning process and quality outcomes. Many factors related to the structured learning process, organization, educator, student, and psychosocial domains determine the quality of overall learning experience in the OR setting. Previously, mini-STEEM, an abbreviated version of STEEM, has been used to assess the OR learning process's quality. However, its content validity has been challenged recently, considering expanding dimensions of the OR learning process.

There is a growing body of literature that directs and indicates the expanding role of a structured learning process despite the difficulties of its implementation within the OR setting. The structured learning process involves careful course planning, delivery process, and assessment involving the psychomotor and affective components apart from the operative learning's cognitive portion. Similarly, educator-related and student-related factors significantly influence the learning process within the OR setting. Teacher's interest, competence, style, and welcoming attitude are independent predictors of quality OR learning. Faculty training in this regard can be pivotal and also provides chances for better student handling within the OR setting. The students' interest, receptive attitude, capacity to self-regulate learning and social handling within the OR can alter the outcomes of the learning process. The organizational support in terms of providing adequate technology and administrative support has additive value. Psychosocial training of the faculty and the students can positively affect the quality of learning process.

References

  1. , , (). Factors that influence medical student learning in the operating room. Medical Teacher 41(5), 555-560 https://doi.org/10.1080/0142159X.2018.1504163
  2. , , (). The operating theatre as a classroom: a literature review of medical student learning in the theatre environment. International Journal of Medical Education 10, 75-87 https://doi.org/10.5116/ijme.5ca7.afd1
  3. (). Making the most of learning in the operating theatre: student strategies and curricular initiatives. Medical Education 37(8), 680-688 https://doi.org/10.1046/j.1365-2923.2003.01583.x
  4. , , , (). The briefing, intraoperative teaching, debriefing model for teaching in the operating room. Journal of the American College of Surgeons 208(2), 299-303 https://doi.org/10.1016/j.jamcollsurg.2008.10.024
  5. , , , Students' perception of the educational environment in the operating theatre: development and psychometric evaluation of a questionnaire German Medical Science GMS Publishing House. DocO13 https://doi.org/10.3205/17rime13
  6. (). Development of an instrument to measure the surgical operating theatre learning environment as perceived by basic surgical trainees. Medical Teacher 26(3), 260-264 https://doi.org/10.1080/0142159042000191975
  7. , , , , (). Perceptions, training experiences, and preferences of surgical residents toward laparoscopic simulation training: a resident survey. Journal of Surgical Education 71(5), 727-733 https://doi.org/10.1016/j.jsurg.2014.01.006
  8. (). A model of teaching and learning in the operating theatre. Medical Education 38(12), 1278-1287 https://doi.org/10.1111/j.1365-2929.2004.02020.x
  9. , , (). Twelve tips for medical students to maximise learning in theatre. Medical Teacher 37(1), 34-40 https://doi.org/10.3109/0142159X.2014.932899
  10. (). ABC of Content Validation and Content Validity Index Calculation. Education in Medicine Journal 11, 49-54 https://doi.org/10.21315/eimj2019.11.2.6
  11. , , , (). Exploring Operating Room-Based Student Learning Experience: Perils & Pitfalls? A Narrative Literature Review. Archives of Surgical Research 1(1), 38-46 https://doi.org/10.48111/2020.01.08