Accountability Measures 2022

For the 2020-2021 cycle, the EPP used data from the Completers Visit to the Classroom Assessment Instrument, a Focus Group for a Completers Interview on Teaching Effectiveness and P-12 Students Impact, the Teachers’ Sense of Self Efficacy Short Scale results, Completers Satisfaction Survey, and a new instrument: Principal’s Teachers’ Evaluation Rubric. This last instrument was developed to standardized principal’s evaluations regarding completers’ effectiveness and impact on P-12 learning and development. The previous year, the EPP asked for completers principals’ evaluations, and all used different criteria.

The EPP has no access to state or private systems student growth data. The information is available through general school statistics that do not demonstrate completers’ contributions to P-12 learning and development. The EPP is using program information from multiple measures to demonstrate that completers are having a positive impact on P-12 students.

Completer Visit to the Classroom Assessment Instrument

The EPP coordinator visited eight completers (29.6%) at their workplace and observed them for a teaching period. Table 1.0 shows completers demographics, areas of specialization, and workplace category (private/public).

The Completer Visit to the Classroom Assessment instrument consists of 21 items aligned to the InTASC Standards for Effective Teaching. The instrument scale is divided in four stages: Distinguished (4); Proficient (3); Emerging (2); and Underdeveloped (1). In addition, the instrument has three in-between stages: Partial success at rating Distinguished (3.5); Partial success at rating Proficient (2.5); and partial success at rating Emerging (1.5). Table 1.1 shows participants scores on their performance teaching their P-12 students.

Mean scores are representative of what these completers experienced as they moved from emergency virtual, to hybrid, and to in-person teaching. 75% of completers perfromed in the proficient or higher category and 25% scored at partial success at rating proficient. The higher mean scores were obtained on InTASC standards 4, 6, 7, and part of 8. These InTASC standards are related to the use of teaching strategies, communication skills, planning, and assessment techniques. This group of completers did their practicum virtually and had to use a new repertoire of strategies and techniques to maintain P-12 students engaged. As schools moved to in person teaching, completers took all the virtual experiences to the classroom and enriched group dynamics. Lowest scores were obtained on InTASC standards 1, 2, and 10. The EPP did a similar analysis for the strengths of these scores. These completers started as novice teachers in a virtual environment. Much attention was given to student engagement and attention rather than the study of central concepts or building relationships with other members of the school community. In addition, completers C6, C7, and C8 are from elementary and preschool education. In these levels, some completers strived to have children engaged through virtuality which affected subject matter teaching. There are no definite trends between 2018, 2019, and 2020 years. The instrument was used for the first time for the 2019-2020 cycle of evidence. The items of concern were related to InTASC 3, 9, and 10. Changes in educational modalities have altered the expected patterns of results. Nevertheless, there seems to be a common denominator: InTASC standard 10 The teacher fosters relationships with school colleagues, parents, and agencies in the larger community to support students’ learning and well-being. The EPP is working on modifications to the courses EDUC 3015 Clinical Experiences I and EDUC 4013 Clinical Experiences II to allow candidates opportunities to collaborate and support students, parents, and outreach community partners to strengthen school relationships and to enrich the learning environment.

Completers Focus Groups on Teaching Effectiveness and P-12 Students Impact

The EPP invited all 2020-2021 completers through emails to a Focus Group Interview on March 4^th, 2021, at 7:00 pm using ZOOM Platform Meeting ID 988 146 8462, access code 840122. Five out of twenty-seven completers (18.51%) participated in the focus group interview. The structured interview consisted of eleven open-ended questions. The Focus Group was conducted by a trained counselor who is not part of the EPP staff. The EPP coded and categorized the qualitative data that derived in various themes that were used to analyze these data. Table 1.3 shows the central themes derived from each question of the interview.

Completers answers gave the EPP a broader perspective of the school scenarios, working conditions, and what is needed to be taught to prepare confident and knowledgeable teachers. The EPP understands that novice teachers struggled with certain administrative and teaching aspects like using the assessment cycle to re-teach, being confident with classroom management skills, differentiated instruction knowledge and strategies, and having few direct teaching experiences throughout their preparation. In this matter, the EPP has developed new key assessments for candidates to strengthen the program and their teaching skills (See Standard 5 Quality Assurance System). In addition, the Teacher Preparation Program is under revision by all the Interamerican campuses that offer the program and that provides the opportunity to discuss the need for system changes. This is the first year that the EPP uses the Focus Group Interview. For the cycle 2019-2020, the EPP interviewed completers on Teaching Effectiveness and P-12 Students’ Impact by phone. Data reflected that completer were using the assessment cycle to revise planning and to use a variety of teaching strategies to approach each learner needs. These findings are completely different from the 2020-2021 focus groups. There are no trends between the two cycles of data.

Teachers’ Sense of Self-Efficacy Scale
The Teachers’ Sense of Self-Efficacy Short Scale assesses teachers’ capability concerning instructional strategies, student engagement, and classroom management. The instrument uses 9 points rating scales: None (1) and (2), Very Little (3) and (4), Some degree (5) and (6), Quite a Bit (7) and (8), and A Great Deal (9). The EPP used it last year to support the Completers Case Study. For the 2020-2021 cycle, the scale was administered to twelve out of twenty-seven completers (44.44%). The scale distributes questions according to Efficacy in Student Engagement items 2, 4, 7, and 11; Efficacy in Instructional Strategies items 5, 9, 10, and 12; and Efficacy in Classroom Management items 1, 3, 6, and 8. Tables 1.4, 1.5. and 1.6 show the distribution of items and disaggregated scores. Table 1.7 shows The Teachers’ Sense of Self-Efficacy Short Scale aggregated mean scores and standard deviations.

Data from the Teachers’ Sense of Self-Efficacy Short Scale results demonstrated that completers perceived the use of diverse and appropriate instructional strategies as an area of strength and that classroom management skills as a weaker point to their teaching. These results are consistent with the Completer Classroom Visit Assessment and the Focus Groups Interview findings. For the 2019-2020 cycle, the EPP administered the instrument to three completers that were part of the case-study. Last-year findings (mean scores 7.53, 7.53, 6.90) were higher for student engagement items and for instructional strategies, but lower for classroom management items. Nevertheless, there are not significant changes between cycles of data. The trends are that completers perceived that they have the confidence and self-control needed to produce gains in P-12 students learning.

Completers’ Satisfaction Survey
The Completer Satisfaction Survey was administered to the twelve (44.44%) out of 27 completers that answered the Teachers’ Sense of Self-Efficacy Short Scale. The demographics and specializations are the same as portrayed in Tables 1.4, 1.5 and 1.6. The completer satisfaction instrument is a 17 questions instrument with a 4 criteria Likert-scale; Well-prepared (1); Sufficiently Prepared (2); Not Sufficiently Prepared (3); Not Prepared at All (4). The survey is intended to inquiry how satisfied were completers with the EPP preparation. Completers’ satisfaction and sense of self-efficacy are often related and imply commitment to perform diligently. Table 1.8 shows Completers’ Satisfaction Survey disaggregated scores

Overall (58%) of completers felt well-prepared or sufficiently prepared as teachers, and 42% felt between not sufficiently prepared and not prepared. These completers did most of their preparation under the stress and uncertainty that were caused by hurricane Maria, the earthquakes that affected the island, and the Covid-19 pandemic. The natural disasters affected both, faculty, and candidates. In addition, the transition to virtual instruction was a major source of stress to a faculty that was used to in-person education. All these factors contributed to a state of discomfort and frustration. Nevertheless, EPP administrators started retraining faculty in distance education and courses have been modified to satisfy candidates needs and demands. For the 2019-2020 cycle, completers scores were higher except for the Special Education specialization. This cycle, lower scores were reflected for ESL completers. There are no trends for completers satisfaction as these findings responded to a new “emergency” reality.

Principal’s Teachers’ Evaluation Rubric
The Principal’s Teachers’ Evaluation Rubric is a new instrument that the EPP piloted with the 2020-2021 completers. The instrument was developed to measure teachers’ competencies from a direct supervisor, other than the employer. In addition, the EPP wanted to use this information as an indirect measure of P-12 students’ growth. State statistics on public schools are limited to general information concerning core subjects and are not distributed by teacher. The same is true for the private educational system. This educational system uses a private company, and the information cannot be shared to particulars. The rubric is divided into the eight competencies that are expected for effective teachers: Communication and Interpersonal Skills; Organization and Planning; Classroom Management; Facilitation and number of; Assessment and Coaching; Collaboration and Teamwork; Caring and Inclusiveness; and Flexibility and Adaptability. It uses a three-point scale High Competency (3), Distinguished Proficient (2), and Emergent (1). This instrument was validated by ten judges. Validation results was 0.0884 , medium validity, using Validity Index of Aiken. The instrument was completed at the EPP visit for Completers’ Classroom Observation Assessment by their direct supervisors. Table 1.9 shows direct supervisors scores for completers using the pilot instrument.

Completers lower scores were related to Organization and Planning, Classroom Management, and Assessment and Coaching. Completers strengths were demonstrated at Collaboration and Teamwork, Caring and Inclusiveness, and Flexibility and Adaptability. These findings are consistent with the ones from other instruments. The 2020-2021 cycle completers represented an educational transition that was marked by an emphasis on socio-emotional development and care rather than on academic rigor. Since this instrument was piloted for the first time during the 2020-2021 cycle and for a small group of novice teachers, there are no data for comparisons.

In conclusion, for Measure 1 the EPP used multiple instruments to demonstrate that 2020-2021 program completers effectively contributed to P-12 student growth learning and applied the professional knowledge, skills, and dispositions required for a highly competent teacher.

For the 2020-2021 cycle, the EPP used the Employers’ Satisfaction with Completer Survey to assess completers’ mastery of teaching and professional competencies as perceived by the employer and the Principals Teachers’ Evaluation Rubric to assess completers teaching competencies from a direct supervisor. In addition, the EPP included stakeholders’ co-construction efforts for the Principal’s Teachers’ Evaluation Rubric, which is in a pilot phase. The EPP administered both instruments on site visits. Eight employers (29.6%) and principals (29.6%) or teacher direct supervisors completed the instruments. The Employers’ Satisfaction with Completer Survey was used last year for the first time. The new, co-constructed instrument, Principal’s Teachers’ Evaluation Rubric, was developed to uniform schoolteachers’ evaluations on teaching competencies aligned to best practices, InTASC Standards, CAEP, and Puerto Rico Department of Education Professional Standards.

The Employers’ Satisfaction with Completer Survey uses a five-point Likert Scale and has 10 items. The Likert scale goes from Strongly Agree (5); Somewhat Agree (4); Somewhat Disagree (3); Strongly Disagree (2); No Response (1). Mean scores were higher than last year with an average mean of 4.7 out of 5. The lower scores (mean=4.2) were on questions #2 and #4 related to providing learning opportunities to support children’s intellectual, social, and personal development, and in using a variety of instructional strategies to encourage critical thinking, problem solving, and performance skills. There are no trends between the years 2019-2020 and 2020-2021. For the 2019-2020 cycle, the EPP used the Employers’ Satisfaction with Completers Survey for the first time and findings were related to completers use of virtual platforms since it was the first year of systemic virtual education in Puerto Rico. Nevertheless, employers’ evaluation scores have been consistent between (5) Strongly Agree, and (4) Somewhat Agree for the last two cycles.

These cohort of completers did their in-service experience virtually and returned to hybrid and in-person instruction for the 2020-2021 cycle. Some completers were still adapting to a group of children that had been secluded at home for almost two years. In fact, schools were trying to figure out what were the best strategies to positively impact P-12 students. Despite all the adaptations, employers were satisfied with completers achievements. Table 2.0 shows completers demographics, area of specialization, and school system of education where they are teaching. Table 2.1 shows Employers’ Satisfaction with Completers disaggregated scores.

Other than employers’ satisfaction, the EPP wanted specific information on completers competencies from their principal or direct supervisor to complement the employers’ feedback on performance, and for its continuous improvement plan. The EPP acknowledged that this new instrument has meaningful information that could serve to strengthen candidates’ preparation. The EPP invited a group of three principals, from the pool of collaborative in-service schools to co-construct an instrument for the purpose of measuring teachers’ competencies (See Appendix 2.1). After three meetings and hours of searching through instructional web pages, the EPP and stakeholders agreed on using the teachers’ competencies framework (European Commission, 2013) to develop a three-point scale rubric: High Competency (3); Distinguished Performance (2); Emergent Competencies (1). The instrument assesses eight areas of teacher competency: Communication and Interpersonal Skills; Organization and Planning; Classroom Management; Facilitation and Engagement; Assessment and Coaching; Collaboration and Teamwork; Caring and Inclusiveness; and Flexibility and Adaptability (See Appendix 2.2: Principal’s Teachers’ Evaluation Rubric and content validation indexes). Table 2.2 shows school principals and/or director’s evaluations for the eight completers.

Other than employers’ satisfaction, the EPP wanted specific information on completers competencies from their principal or direct supervisor to complement the employers’ feedback on performance, and for its continuous improvement plan. The EPP acknowledged that this new instrument has meaningful information that could serve to strengthen candidates’ preparation. The EPP invited a group of three principals, from the pool of collaborative in-service schools to co-construct an instrument for the purpose of measuring teachers’ competencies (See Appendix 2.1). After three meetings and hours of searching through instructional web pages, the EPP and stakeholders agreed on using the teachers’ competencies framework (European Commission, 2013) to develop a three-point scale rubric: High Competency (3); Distinguished Performance (2); Emergent Competencies (1). The instrument assesses eight areas of teacher competency: Communication and Interpersonal Skills; Organization and Planning; Classroom Management; Facilitation and Engagement; Assessment and Coaching; Collaboration and Teamwork; Caring and Inclusiveness; and Flexibility and Adaptability (See Appendix 2.2: Principal’s Teachers’ Evaluation Rubric and content validation indexes). Table 2.2 shows school principals and/or director’s evaluations for the eight completers.

Findings suggests that items 3 Classroom Management, 4 Facilitation and Engagement, and Assessment and Coaching were closer to the emergent score. In general, completers were rated as Distinguished Performance which is good for novice teachers. Nevertheless, this information confirmed what other instruments findings suggested regarding classroom management and using assessments. The EPP will meet during the summer to discuss findings and to develop concrete activities to enhance these skills in candidates. In addition, the EPP will meet with co-constructors to refine the instrument and to add criteria to the Emergent column which was not defined. The instrument will be piloted with the new structure for the next cycle of completers.

EPP completers are often hired as candidates during their internship or before. Puerto Rico had been experimenting a shortage of teachers and employers are always seeking highly-qualify candidates to recruit. In addition, school districts from the United States are constantly visiting the campus for recruitment fairs. The 2020-2021 completers were the first group to return to in-person education after the pandemic seclusion. Table 4.1 shows August-December 2020 completers working status disaggregated by academic level and academic system (public/private). Table 4.2 shows January-May 2021 completers working status disaggregated by academic level and academic system (public/private).

For August-December 2020 cycle, six (50%) completers were working at their teaching specialization level in private schools, five (41.66%) were at public schools, one (.08%) was working in the Continental US School system, and the EPP could not establish communication with one. For the January-May 2021cycle, seven (46.6%) were working at their specialization level in private school, four (26.66%) were working in public schools, one (.066%) was working in the Continental US School system, and three (20%) were not working at all. These findings demonstrated that EPP completers have all the competencies required to be hired in education positions for which they have been prepared.