Assignment #2

Prenatal Exercise Program for Aboriginal Women à Evaluation Proposal

One of the main difficulties in selecting an evaluation approach is the lack of information provided in the case study. The document does open with a clear explanation of why an exercise program for pregnant Aboriginal women in Saskatoon is an excellent idea. However, the provided description never really states the goals of the program beyond the idea that “initiatives to promote regular physical activity among Aboriginal women during their childbearing years could play a major role in optimizing healthy pregnancies and in reducing rates of type 2 diabetes in future generations”. It also becomes very difficult to evaluate a program without knowing the future intent of the program (summative or formative perspectives).

I think a naturalistic perspective is the logical starting point for this program evaluation. It appears that a major component of the program was the recruitment and retention of the targeted demographic. The program developers began by seeking partnerships with the Aboriginal community and then looked for support from an Aboriginal Project Facilitator and the advice of an Elder. The case study seems to focus a great deal of attention on the participants and not the medical statistics. In fact, the majority of statistical analysis appears to be about the rate of participation and not the rate of type 2 diabetes.

The evaluation should focus on qualitative data (some quantitative would also be useful) collected through conversations and simple surveys with the 61 participants (the quality of feedback would be much stronger through a discussion group but the anonymous distribution of a survey also has some merit). If possible, it would also be very helpful to acquire feedback from the 45 individuals that inquired about the program but chose not participate. A third set of data from the individuals that began the program but dropped out would also be important (possibly the most valuable) to the evaluation. The evaluation would focus on the three data groups and their perceptions of the program and the implementing individuals. A fourth set of data from the implementers would also provide another perspective to include in the evaluation.

The project developers did not simply create a physical exercise regiment, apply this to a test group, and see if the diabetes ratio in Aboriginal women decreased. I get the feeling that the project developers viewed health through a holistic lens (as does traditional Aboriginal thinking). This was evident through the social components and the health education portion of the program. The program evaluation must include more than the rate of diabetes, it must look at the whole person and examine their physical, emotional and spiritual well-being. These areas are very difficult to analyze in a purely statistical method and this lends further weight to the notion of a naturalistic evaluation.

As this program is likely grant-based, Provus’ cost-benefit analysis could be important for some stakeholders. As an evaluator, I would caution against the notion that we are dealing with a measurable output. The program appears to be about more than simple diabetes rates.

As mentioned earlier, the case study does not clearly state goals and therefore I would recommend a Scriven, Goal-free analysis. This would allow for an external, unbiased and naturalistic perspective that simply allows the evaluators safe entry to the thoughts and feelings of the participants and program implementers.

Some key questions:

-          What was the role of the Elder and how did they influence the program and participants?

-          What was the role of the Aboriginal Project Facilitator and how did they influence the program and participants?

-          How did the Aboriginal community feel about the program and participants?

-          What was the role of program implementers and how did they influence the program and participants?

-          Was the level of exercise (duration and intensity) enough to have a medically relevant impact?

-          How did the program function in terms of a means and an end to address health issues?

-          Should the program be implemented again?

-          Can the recruitment plan be improved (only 7% participation)?

-          What impact did the facility have?

-          How can all demographics be reached?

o   Why did specific demographics participate?

-          Were there any other health benefits?

-          Was there an increase in health knowledge/education?

-          Was there an emotional or spiritual impact felt by participants?

-          Would participants return if they became pregnant again?

-          Did the participant get diabetes?

-          What impact did the social area have?

-          Etc.

Assignment #1

Program Overview:

4 For Lunch is a four-week curriculum-matched program culminating in a challenge week where students are to bring healthy lunches containing four food groups. As part of the program, teachers received lessons meeting the outcomes of the Ontario Health and Physical Education Curriculum. Parents were also provided with nutrition resources that aligned with the program. The main goal was to encourage children and their families to pack healthy lunches. The program believes that by enhancing the students’ self-efficacy through the classroom-based intervention, the confidence will be transferred to other nutrition behaviors. The hope being that students will have the skills, knowledge and self-efficacy needed to make lasting behavior changes.

Evaluation Process:

The first task of the evaluators was to understand the purpose of the program. This knowledge was then used to generate four research questions that guided the rest of the evaluation. The evaluators also engaged in a literature review to help them identify the traits highlighting successful public health interventions in nutrition. The evaluation was summative in nature as it only analyzed the current program results in an isolated one-month snapshot. The evaluation could also be perceived as formative because the recommendations will likely play a key role in ‘tweaking’ the program for future implementations. The evaluation was experimental in design as it looked to compare data from implementing and control groups in order to determine if the program was meeting its goals. Participants for the study were gathered through flyers and letters to all elementary schools in the region. Proper methodological /ethical considerations and permissions were undertaken and approved by the school boards prior to commencing the study. To determine the success of the 4 For Lunch program, numeric data was collected via the previously validated pencil/paper pre- and post-test measurements tools of: teaching self-efficacy; student nutrition knowledge; and 24-hour student food recall questionnaires. The findings were shared in a formal report. This report included a thorough discussion of results, while maintaining a focus on the four guiding questions. It also identified limitations of the evaluation and finally provided key recommendations for future implementations of the program. The evaluation had a similar feel to Stake’s Countenance model as it clearly used descriptive data searching for congruence between the intended and what is actually observed.

Strength / Weaknesses of Evaluation:

I thought the evaluation did a thorough job of uncovering and highlighting the theory behind the program’s implementation. I also felt the literature review provided an academic component that gave the reader an immediate confidence in the findings. Reader confidence was also gained by the formal and well-organized appearance of the final evaluation report. The evaluation effectively used tables and charts to help convey the findings to the reader without getting lost in the technical aspects of the statistical analysis.

The discussion worked very hard to justify the lack of positive numeric results as shown by the collected data. It quickly became clear that proponents of the program commissioned the evaluation. Realizing that this was likely the reason for completing the evaluation, my criticism lies with the overt lack of neutrality, not the fact that bias entered the discussion portion of the evaluation (the hidden curriculum of program evaluation was clearly evident). I believe the evaluation should have spent more time pointing out that the major breakdowns occurred in the flawed implementation (lack of training and consistency). The limitations section of the report highlighted the majority of the concerns that I was going to raise, which was excellent, but I felt that they should have taken this one-step further and identified how these limitations may have affected the results.

Overall, I would say that this program evaluation was well-orchestrated and the difficulties did not lie in the evaluation strategy but more in the fact that it was trying to cover a flawed program.

Here is a link to the Program Evaluation I chose to analyse:

http://www.york.ca/NR/rdonlyres/y4ev2ss7botporw46bvivc3rrgnviwyry3visaabh72fkuhwurnwh6mazhf5d2ktpffyqvlxljgi7af2lschwpulre/4_for_Lunch_Evaluation_Report.pdf

It takes a look at the "4 for Lunch" program being utilized in 42 Ontario schools.

Test Run

Hello ECUR 809 Class....