spacer

monitoring and evaluation

IMPACT

The specific goal of each presentation is that, through

  • the encouragement of a positive self-concept,
  • the enforcement of self-esteem and self-value,
  • the provision of relevant information
  • and the development of decision making life-skills,

the learners will be enabled to

  • make sound, informed lifestyle choices about what is best for them, and their community, in their particular contexts;
  • accept and take responsibility for those decisions;
  • be flexible and robust in responding to the consequences, both foreseen and unforeseen.
Clarence Crocodile tries to eat Tommy Turtle in a scene from No Monkey Business: Us and Them. Gauteng 2003
spacer
Clarence Crocodile tries to eat Tommy Turtle in a scene from No Monkey Business: Us and Them. Gauteng 2003
spacer

They are resilient and enabled with informed choice for sustainable decision-making.

In the context, impact is therefore specifically understood as giving to, or affirming, or verifying for the audiences the skills, information and confidence to a) make and validate their own choices, b) take responsibility for those decisions, and c) to be adaptable, open and flexible in responding to the consequences, and circumstances in general.

For an presentation to have this impact three factors are essential; personal emotional identification by the audience with the performance, the provision of relevant factual information, and the promotion of cognitive discourse amongst the audience.

defines these factors with the following Engagement Indicators:

Information: The learners have been (re)informed with relevant factual information.
Identification: The learners personally identified with the performance.
Cognition: The learners cognitively engaged in discussion and debate about their issues.

Identification in this context is understood as a powerful feeling of affinity with the characters and their situations, and involves potentially regarding the characters as models that they may recognise, aspire towards, or avoid; adopting, realising or rejecting their attitudes, beliefs, behaviours, lifestyles or views.

Cognition is the mental ability and process of acquiring knowledge regarding attitudes, beliefs, behaviours, lifestyles and views by the use of reasoning, intuition or perception.

uses these engagement indicators to determine the depth and level to which they have involved the audience in the presentation. The better the presentation succeeds in engaging the audience on these three levels, the better the environment for impact to occur amongst that audience. The impact that is expecting is in terms of the audiences’ self-concept, self-efficacy and sustainability in relation to the specific issues that the audience engaged with during the intervention.

Self-concept in this context is understood to be the whole inner picture that somebody has of himself or herself, including such traits as competence, worth, and attractiveness - how they think, believe and feel about themselves.

Self-efficacy is understood to be the ability of a person to understand, influence and change their own attitudes, beliefs, behaviours and views.

Sustainability is understood as the maintenance of a person’s self-concept and their choices, beliefs, behaviours and views, and includes the ability to be tolerant, flexible, and adaptable.

defines this impact with the following Resilience Indicators:

Self-concept

Self-awareness: an increase in the learner’s competence and understanding of themselves, developing self-image – how they think about themselves.

Self-esteem: an increase in the learners' sense of confidence, merit and value – how they feel about themselves.

Self-respect: an increase in the learners’ sense of worth and dignity – their belief in themselves.

Self-efficacy

Choices: an increase in the learners’ understanding about the options and consequences of lifestyle choices, beliefs, behaviours’ and views.

Decisions: an increase in the learners’ ability to make decisions regarding their own lifestyle choices, beliefs, behaviours’ and views.

Responsibility: an increase in the learners’ acceptance of the responsibility for decision-making, or, conversely, for not making decisions regarding their own lifestyle choices, beliefs, behaviours’ and views.

Sustainability

Tolerance & Open-mindedness: an increase in the learners’ tolerance of and receptiveness to alternate choices, beliefs, behaviours’, lifestyles and views.

Flexibility: an increase in the learner’s ability to accommodate or allow for alternate choices, beliefs, behaviours’, lifestyles and views.

Adaptability: an increase in the learners’ ability to adjust and modify to changes and new conditions/circumstances.

uses these indicators to determine how the audience may have been changed by their engagement in the intervention, and in which particular self-development areas. A change does not need to have occurred in all indicators for there to have been an impact, however the greater the level of the impact, the greater the likelihood of the changes being followed-up and maintained, with the appropriate support. refers to this overall impact as Resilience.

Resilience here is understood as the capacity to withstand, transcend, respond to and recover from changes, set-backs or adversity.

The development of self and self-efficacy is not a single event, but rather a process which occurs on a continuum. Impact is therefore understood to occur at various places on this continuum to facilitate change – from a person starting on the continuum towards change, to facilitating movement along the continuum, to assisting in the maintenance of that change – depending on where the person is in relation to the issue and their self-development.

has defined five basic stages on a Self-efficacy Continuum to indicate the nature and depth of the impact that occurred for the audience, based on the manner and level with which they engaged in the specific issues, as follows:

Sparked Awareness is the point where a person begins the process of possible change; the impact is that they become aware where they were not before and start thinking about a choice, a behaviour, a belief, a lifestyle.

Re-enforced Understanding is the point where the information and knowledge that a person has gathered confirms a possible choice about a behaviour, a belief or a lifestyle; the impact is that their understanding is enhanced and re-enforced concerning their options.

Encouraged Choices is the point where a person actually plans to make a change; the impact is that they choose to address a behaviour, a belief, a lifestyle.

Supported Decisions is the point where the person takes action; the impact is that they change the behaviour, belief, lifestyle.

Validated Lifestyles is the point where a person is living and maintaining the change; the impact is that it confirms and supports the changed behaviour, belief, or lifestyle of a person, validating their choices and decisions.

This can be expected to occur in several places due to the diversity of the audience and the nature of the issues presented.

MONITORING AND EVALUATION

spacer
Puppets from No Monkey Business: Inside Out on display at the National Puppetry Exhibition, ABSA Gallery 2003
spacer
Puppets from No Monkey Business: Inside Out on display at the National Puppetry Exhibition, ABSA Gallery 2003

In order to determine whether the desired impact of an presentation has occurred, each performance team is trained to critically assess the learners reactions to the presentation, their participation in the facilitation sessions, the discussions held, and the questions asked, and then, in discussion, to rate and report on their impression of the potential impact.

At each point the quantitative ratings are supplemented and supported by qualitative statements to provide explanation, and to assist with later recall, discussion and reporting.

The intent is to capture the immediate impact of each presentation with each individual audience as assessed by the performance teams. In this way is able to determine where in relationship to the development of resilience each audience is sited with regard to the main issues that were relevant to them from each presentation, and what potential areas may need additional concentration, follow-up or support.

In order to provide a sense of validity to the teams’ perceptions, the Life-Skills educator at each school, and some learners do the same after each presentation, using the same or similar criteria and processes. This provides a three point rating of each presentation by all the participants for comparison and analysis.

This enables to identify, capture and assess qualitatively and quantitatively the impact of the individual presentations, the tours they form part of, and the audiences they perform to.

These ratings and reports are then tracked and compared with the situations, environments and profiles gathered from other presentations and other schools, enabling , over time, to statistically demonstrate change and the potential for sustainability in these areas and thus the impact of the project.

The arepp:Theatre for Life M&E System

In order to determine whether there could have been impact, and the level of that impact, the Engagement Indicators - Information, Identification and Cognition - are first assessed, with qualitative examples to support each decision, on a 6 point scale from negligible to considerable, where negligible has a value of 0 and considerable a value of 5.

Considerable 80%+, Significant 60%+, Reasonable 50%, Some 40%+, Minimal 20%+, Negligible 20%-

The three separate ratings are then combined and divided by 3 to provide an Engagement Rating out of a possible maximum of 5. This then indicates the depth and level to which that audience, as a whole, engaged with the presentation as a whole - The higher the rating, the greater the depth and level of the audiences’ personal involvement and identification with the presentation.

The engagement rating is then further expressed in terms of the percentage of the audience which gave rise to it. This overall Engagement Rating is then supported with a short narrative explanation of how and why the team made those decisions.

After this analysis of the audience’s level of engagement, the team then determines what the main issues where for that audience, and what the particular elements/aspects of those issues were. These are then listed, with an attendant narrative summery of the salient points relating to each issue into Issue Statements.

Each of these main issue statements and their nature is then examined in relation to each of the Resilience Indicators. After discussion, with qualitative examples to support each decision, the team rates the possibility of change in each indicator on a 5 point agreement scale from disagree to definitely, where disagree has a value of 1 and definitely has a value of 5.

Definitely, Agree, Possibly, Not Sure, Disagree

Each of the indicators in each of the three areas - self-concept, self-efficacy and sustainability – is then combined and divided by 3 to give a potential Impact Rating in each of the three impact areas, out of a possible maximum of 5. Each of these is then supported with a short narrative explanation of how and why the team made those decisions.

This indicates in what ways the audience may have been changed by their engagement in the intervention and in which particular self-development areas. It also indicates which areas may need follow-up and support .

The three impact ratings are then combined and divided by 3 to give a Resilience Rating, out of a possible maximum of 5. The resilience rating is then further defined in terms of where on the self-efficacy continuum and to what level this impact was perceived to have occurred.

Sparked Awareness, Re-enforced Understanding, Encouraged Choices, Supported Decisions, Validated Lifestyles

These overall Resilience & Self-efficacy Ratings give an overall impression of the possible impact and its sustainability, and are supported with a short narrative explanation of how and why the team made those decisions.

This, together with the Engagement Rating then gives an overall quantitative expression of the intervention, which is supported and enhanced by the combined qualitative narrative sections.

The aim is to express a ‘snap-shot’ of the possible impact of each presentation in terms of an analysis of the status, level and nature of the engagement and interaction of that audience with the intervention, in order to gain an impression of the sustainability of that impact and an indication of the support and follow-up necessary to assist that sustainability.

© 2006 arepp: Theatre for Life | Designed by idDigital