Standard Indicators for Measure of Impact on Hygiene Behavior

Objective:  to identify a set of hygiene behavior indicators which might be
commonly used by all KAP (knowledge and practice) surveys in Myanmar.  Six have
been tentatively identified so far.  They measure two general areas of
intervention, hand washing and latrine use.

To sort out terminology, for the duration of this discussion,
"monitoring" will refer to a measurement taken for purposes of managing
operations (process indicators). 
"program evaluation" will refer to evaluation intended to discover how to adjust
or modify a program.
"measure of impact" refers to a measure of progress toward the ultimate goal or
major objectives of the program

Enclosed is an article, from a WELL FACT SHEET, 
Measuring the health impact of water and sanitation, by Sandy Cairncross.  It
is included here because it talks about the reason for resorting to hygiene
behavior indicators for measure of impact, rather than using measures of
reduction in diarrheal disease.  In short it says that measures of impact on
disease is difficult to make, and furthermore, that when they can measure
success/failure, they offer no explanation of what aspects of an intervention
worked or did not.  

It is important to remember, when reading that helpful article, that the measure
of impact on disease is still widely practiced, and has been of great value in
guiding water and sanitation interventions.  It is because of those measurements
that it was possible to discover that sanitation and hand washing interventions
actually make a greater impact on disease than water quantity and water quality
interventions.  (Also enclosed, please see the chart "Effectiveness of Hardware
and Hygiene Interventions in Reducing Diarrhea Morbidity" from a World Bank web
site called "Water and Sanitation at a Glance")  As a result of this realization
it is widely accepted, today, that water without hygiene education is
ineffective.   Furthermore, it is important to recognize that the article is
not saying that the reduction of disease is not the ultimate goal of water
and sanitation interventions.

Still, the Cairncross article explains that measuring changes in hygiene behavior
has value for program evaluation (for purposes of program adjustment) and also
for measures of program impact.  Discussions here are limited to measures of
impact. Program evaluation can be conducted more frequently and less rigorously
than measures of impact.  Furthermore, standardizing indicators across
different NGO)programs has its greatest value when measuring impact.  Standard
indicators may not be so important for internal adjustments or modifications of
individual programs.

Attached is a table of indicators.  The format,
rather than the contents, is the most important aspect of the table; because
it can provide a tool with which standard indicators can be selected.  The
columns are described below.  The first six rows are indicators that were chosen
for common use by a group of NGOs.  The remaining rows, as you can see, contain
indicators which do and others which do not meet the criteria. 

Column 2,  Aim:
"Aim" refers to the program message that the indicator aims to (is meant to)
 measure.  This may be the most important criteria, because, of course, the
 indicator must match programs very closely.  For example, if a program uses
 "clean fingernails" as an indicator, the message would have to have included
 cleaning nails, not simply washing hands.  

Some programs carry more messages than others.  Some focus on hand washing and
latrine use, while others might also include water handling, general sanitation,
even water filtering etc. Most of the standard indicators should be applicable
to all programs, but it would be possible to include some indicators that can
only be used for some programs. Each impact survey could simply include as many
of the standard indicators as possible.

It might be useful to make a list of the program messages that should be
included in the indicator list.  It's important to stress that organizations do
not have to agree to include all these messages.  They only need to agree to
use the standard indicators which apply to their programs.

Example program messages:
latrine use 
latrine maintenance (sanitary latrines)
hand washing 
soap use (this, for example, is not the message of some programs)
water hygiene 
food hygiene  

Column 3, Data Collector:
While all indicators could be collected by an external surveyor, we identified
two which could be collected by a member of the community if that person were
trained to do so.  We also discussed the possibility of other indicators which
collected by community members without any special training.  

While external surveyors might be less biased, they might also be viewed as
more intrusive.  It is also possible, however, that external surveyors could be
viewed as less intrusive, because it may be more comfortable for a woman to
report hygiene habits to a professional (who is promising confidentiality) than
it would be for her to report to a neighbor who is collecting data.  

Having community members collect data is a very participatory approach.  That
approach is popular among NGOs, so it might be possible to for organizations to
agree on a standard set of indicators, but collect the data in different ways. 

Column 4,  Preferred Method of Collection:
While all indicators could be collected through interview question, some could
also be collected by observation, offering the advantage of speed, and
sometimes,greater accuracy.  We also discussed the possibility of other
indicators which collected through demonstration.

Column 5, Level of Data:
Some interventions take place at the community (village) level, for example
water point maintenance.  Others, could be measured at the individual level, for
example, the percentage of children under 5 that use a latrine.  Most of the
Wat San interventions, though, can be measured at the household level.  

Individual level indicators could also be measured as hh level indicators.  For
example, rather than measuring the percentage of children under five that use
the latrine, it would be possible to measure the percentage of households with
children under five in which the children use the latrine.  That would mean
that, if there were two children under five using a hh latrine, the data would
record "yes-1(household)", rather than "yes-2 children".  If the target of the
behavior change programming is mothers, the household level data might be the
best choice. If school latrines are a part of programming, the individual level
data might be the best choice.

There is another consideration to keep in mind when collecting household level
data.  If the necessary sample size is calculated to be, for example, 500, it is
possible that only 150 households will have children under five.   That would
mean that the sample size for that particular indicator would be too small to
give the desired level of confidence in the measurement.   Obviously this can
be corrected by increasing the sample size accordingly.  This may or may not be
considered important, but it should be discussed at least briefly.

Column 6, Can be collected at the same time of  year:
If  an indicator had to be collected at one particular time of year, and others
could not be collected in the same season, data collection would be slow and

Because many behaviors fluctuate by season, it would be important to try to also
standardize (among organizations) the timing of data collection.  Otherwise, if
one program measures latrine use in rainy season, and another measures latrine
use in dry season, their results will be different.  That difference may be
due to the season rather than the program.  That is confounding.  

Column 7, Suitable to Measure Impact:
Many of the indicators on our KAP surveys are useful for running or adjusting
programs.  Counts and measurements of activities are useful for monitoring, but
not for measurement of change in behavior.  Training of personnel, for example,
is essential, but it is not a change in behavior of the target population.
Impact indicators must measure progress toward the ultimate goal or major
objectives of the program's target population.

Column 8, Confounding and Other Difficulties:
Confounding means an indicator is inadvertently  measuring two things, when it
is intended to measure only one.  The second factor, inadvertently included,
may increase or decrease the number.  One example is the clean latrine pan.  The
pan could be clean because the latrine is well maintained, or because it is not
used.  The clean pan can still be used as an indicator, but care would have to
be taken that surveyors also confirm whether or not a latrine is in use. 

Other difficulties might include feasibility.  For example, it might be difficult
to get an accurate measure of the quantity of soap sold in a village. Another
difficulty,  of course, is "reporter bias", the scourge of hygiene interviews L:)

To select more indicators
It might be helpful to borrow more indicators from the experts.  Attached is a
list from the World bank. (Sample Indicators for M&E,  World Bank HE Indicators).  

Frequency of Impact Measurement
After indicators have been chosen, it might be worthwhile to discuss the design
of impact studies.  There are a few issues of interest, for example, how often
impact need be/can be measured.  Enough time must elapse in order for behavior
change to occur, so impact measures can't be too close together.   On the other
hand, if they are too far apart, we wait too long for guidance we need.

Study Design
Another issue is the actual design of an individual program impact study.
Generally speaking, the program population should vary as little as possible
from the population in which the baseline measurement was taken.   

The use of control groups might be considered.  That means doing a base line
measure in a very similar area that will not have the intervention.  That way,
if there are confounding circumstances, for example flood, government HE
campaign, etc, their effect can be subtracted.  The effect of the program can
then be isolated.   

Finally, unless both the indicators and methodology are the same in both the
baseline survey and the impact survey, change cannot be measured.