Program Assessment:
1-6 Guided Reading Groups
As I am part of this organization, it will be important for me to remain as impartial as possible and work hard not to make any assumptions during the planning process. In order to achieve this I will engage in several informal conversations with different stakeholder groups during the initial planning phase. This program is perfect for an evaluation as our staff places a great deal of time and energy into the guided reading groups but are feeling unsure about the impact on student achievement. This is the third year of the reading programs implementation and it has not been evaluated previously. Another reason for performing this evaluation is that recent research is not backing leveled guided reading groups as strongly as it did a few years ago.
The following is a list of key questions (identified by red font) used to reflect and gather information on the guided reading program as the evaluation plan unfolds. It is broken down into several sub-topics as denoted by the bold font.
Engage Stakeholders:
Who should be involved?
- Administration, Teachers, EA’s, Students, Parents, School Community Council (SCC)
How might they be engaged? What are their possible areas of contribution?
- Interviews, surveys/questionnaires, group dialogue sessions, journals, sharing of anecdotal data
o Administration – internal reflection on program goals, resource allocation (personnel and budget), level of implementation, etc.
o Teachers – perspectives on program strengths and weaknesses, suggestions for changes, instructional strategies used within groups, long-term (>2 years) achievement results, thoughts on reading groups vs. daily ELA class time, level of implementation, other testing data, etc.
o Students – personal achievement perspective, time commitment compared to other daily tasks, their engagement in the learning opportunity, etc.
o Parents – level of knowledge of the program, perspective on students’ growth and achievement, etc.
o SCC – perspectives and level of knowledge on the program, etc.
Focus the Evaluation:
What are you going to evaluate? (See attached logic model for clarification of program delivery)
- Is the guided reading program an effective instructional method for increasing students’ reading fluency and comprehension?
- Program Goal:
o Students will continually increase their reading fluency and comprehension levels at an acceptable rate. There are three sub goals depending on their starting level…..
§ Those reading above grade level must maintain (at minimum) their current above grade level
· Ex) A grade two student reading at a grade 4 level must still be at a grade 4 level or higher by the end of the year.
§ Those reading at grade level must maintain (at minimum) their age appropriate increase
· Ex) A grade two student reading at grade 2 level needs to be at grade 3 level by the end of the year.
§ Those reading below grade level must minimize the achievement gap from the start of the year to the end
· Ex) A grade two student reading at a grade 1.3 level (0.7 behind) at the beginning needs to be at a 2.4 level (0.6 behind) by the end of the year when they are going to enter grade 3….keeping in mind that any growth is a success!
What is the purpose of the evaluation?
- Address the accountability of our school goals around fluency and comprehension
- Improve the teaching and learning occurring within the program (and the regular classroom setting)
- Clarify the program goals and progress for all stakeholders
- Inform future decisions about the program:
o What grade levels should be included (i.e. drop the 5/6 class out and add the K’s)?
o Is the time allotment appropriate?
o Do leaders require more training in order to be effective?
o Should additional money be invested into the materials (readers) used?
o Is the primary assessment tool for student achievement accurate enough?
What questions will the evaluation seek to answer?
- Does student engagement correlate with achievement
- Should the program should be continued, altered or disbanded for the 2011/2012 school year?
- Have students been progressing at an acceptable rate (as noted by the three-tiered program goals)
Who will use the evaluation? How will they us it?
- Administration
o To share results and future plans with superintendent at end of year goals conversation
o To dialogue with parents and the SCC about school/program goals
o To inform decisions about staffing/time-tabling/programming and budget allocations for next year
o To facilitate group leaders in reflective dialogue about the program (Professional Development)
- Teachers / EA’s
o To engage with administration and fellow staff about the program (Professional Development)
o To Inform future instructional practice
o To increase confidence in the program or magnify the need/necessity for changes
What information do you need to answer the questions?
- Student Achievement data (pre and post)
- Student and Staff perceptions on levels of learning engagement
- Knowledge of group leader practices (instructional decisions) made during their guided reading group time
o What comprehension / fluency strategies are being consistently employed
When is the evaluation needed by?
- June 2011 in order to allow for post evaluation dialogue and decision making time prior to the beginning of the 2011/2012 school year.
What evaluation design will you use?
- The evaluation will be summative in nature
o If we decide to implement the program in the 2011/2012 school year the current evaluation will become formative as we would conduct another evaluation in June of 2012.
- The design will include a combination of the Provus (Discrepancy) and Stake (Countenance) models
Collect the Information:
What sources of information will you use?
- Existing information:
o Pre and Post-test data from 2008-2009 and 2009-2010 school years
o Pre-test data from current 2010-2011 school year
o Potentially may use data from Prairie View Elementary that is running a similar program
- People:
o Administration, teachers/group leaders, students, parents and SCC
§ Same as above but potentially from Prairie View Elementary
- Pictorial records and observations:
o Sample videos of reading groups
o Administrative observations (with focus on instructional decisions)
o Program manager observations (with focus on student responses to learning situation)
What data collection methods will you use?
- Surveys
- Interviews
- Observations
- Pre/post achievement tests
- Videos
- Journals
§ Due to time constraints (program nearing completion when choice was made to evaluate) these collection methods will only provide a snapshot view. It is recommended that next year collection methods be enacted throughout the program to provide a longitudinal perspective.
What is needed to record the information?
- Instrumentation:
o Surveys
§ Parent
§ Student
§ Group Leader
o Rubrics for both students and group leaders to record levels of engagement
§ Include spot for anecdotal evidence as well
o Interview / group dialogue questions and data response collection template
o Data collection sheet for group observations
§ Instructional focus
§ Student focus
o Video camera computer for playback
o Journal (includes possible reflective questions)
§ This could be utilized by both students and teachers
· Focus would be on the type of strategy utilized
o Reading Assessment for pre/post (Fountas and Pinnell)
When will you collect data for each method chosen?
o Surveys
§ Near end (April-May) / After (June)
o Interview questions and response template
§ Near end (April-May) / After (June)
o Data collection sheet for group observations
§ Near end (April-May)
o Video camera
§ Near end (April-May)
o Journal
§ Near end (April-May) / After (June)
o Reading Assessment
§ Immediately after (June)
Will a sample be used?
- The population is quite small so the majority of collection instruments will be used in their entirety
o Exceptions being: interviews (only use ability cross-section/grade) and video recordings due to time constraints but each group will either be observed or recorded.
Pilot Testing: when, where, how?
- No pilot will be conducted at this time…..however this evaluation and its components could become the pilot for evaluations done in other schools that are running a similar program
Analyze and Interpret:
How will the data be analyzed?
- Data analysis methods:
o Surveys
§ Compiled and assessed for trends in particular fields
o Interviews
§ Compiled and assessed for trends in particular fields
o Video samples
§ Several groups will be filmed and reviewed by evaluators looking for similarities and differences in implementation (instructional decisions). The students in these groups will then be cross-referenced to their assessment scores (note: leader anonymity will be maintained in the report)
· Responsible: Information processing class to film and administration and special education teacher to review
o Group observations
§ This will be utilized in the same manner as the video reviews
· Responsible: Administration / Program manager
o Journals
§ Read by evaluators looking for commonalities and memorable quotes/thoughts (seeking common themes)
· Responsible: Administration
o Reading Assessment
§ Calculate level of increase from the Fountas and Pinnell pre/post assessments
· a) look at average for whole group (1-6)
· b) look at average for individual grades
· c) look at individual students
o Responsible: Special Education teacher / Program manager
How will the information be interpreted – by whom?
- Principal and Project manager will be responsible for making inferences or interpretations of the data. However, staff will be given time to engage in dialogue and identify their own perceptions during the report debrief (teachers will have access to raw data).
What are the limitations?
- Could be a large amount of information to process for only two people (others may need to be brought in)
- There is not time for a pilot test of the collection instruments
- Potentially small sample size
- May have trouble identifying cause-effect relationships due to the large amount of variables present (which can be difficult to isolate)
- Due to the late start of this evaluation, all data is being recorded right at the end of the program, and therefore will only represent a snapshot as opposed to a continuous view of the program
Use the Information:
How will the evaluation be communicated and shared?
- Administration and program manager (special education teacher) will present the findings at a staff meeting
o Raw data and full report will be shared with staff (teachers and EA’s will have full transparency)
o Summary will be presented
§ Open dialogue time with entire staff
o Evaluators recommendations will then be shared with staff
o When: First Monday in June (afterschool meeting)
- Partial information will be released through the principal’s monthly message in the newsletter
o When: Distributed first Friday in June
- Partial information will be released through presentation to SCC
o When: First Monday in June (evening SCC meeting)
- Partial information will be released to students through classroom visits by the administration to let them know about plans for next year.
Next Steps?
- Seek informal feedback from stakeholders after release of findings
- Make final decisions on implementation of program for the 2011/2012 school year
Manage the Evaluation:
- Human subject’s protection
o Students have already signed media release forms to allow for video recording in the school
o Teachers will have option to have their group personally observed by evaluator, fellow teacher or recorded
o Individual student achievement results will not be reported outside of the administration and classroom teachers
- Gantt Chart
o See appended document
- Responsibilities
o Principal – Lead Evaluator
§ Design Evaluation (with support from various stakeholders)
§ Create collection instruments
§ Analyze data
o Special Ed. Teacher – Project manager
§ Pre/Post assessments using Fountas and Pinnell
§ Analyze data
o Teachers/Volunteer Group Leaders
§ Implement the program
§ Participate in surveys, interviews and journaling
o Students
§ Participate to their best ability in program
§ Participate in surveys, interviews and journaling
o Other stakeholders
§ Participate in surveys, interviews and journaling
- Budget
o Use of 4 substitute days
o Cost of photocopying
Standards:
- Utility
o The evaluation will provide valuable information for the administration and program manager when planning for the 2011-2012 school year. This program is an important component to the school culture and learning environment.
- Feasibility
o Due to the small number of active participants and group leaders, the evaluation process should not prove to be too cumbersome to complete. The lead evaluator has the ability to access substitute teacher days should the analysis begin to fall behind.
- Propriety
o This school holds relationships between all stakeholders with high regard. The design should allow for accurate collection and analysis of data within a safe environment. As the lead evaluator is familiar with this learning community, they have made choices to limit any ill feelings from the participants or outside stakeholders. The majority of stakeholders were engaged during the planning portion (participatory or empowerment approach to the evaluation).
- Accuracy
o Project manager (Special Ed. Teacher) will personally perform all pre/post tests to ensure consistent and accurate usage of the assessment tool
§ This individual has formal training in this area
Dean
ReplyDeleteYou have an well-conceived plan for your evaluation. oOu cover all areas extensively and the high level of thought is evident. I agree that this first evaluation will be a good pilot test as well as a summative look at your program. I have nothing to offer you for criticism just 'Good luck'.
Jay