Between a rock and a hard place with literacy rate statistics
by Susan Sussman
When it comes to using literacy rate statistics,
advocates and policy makers are stuck between
a rock and a hard place – we are damned if we
don’t and damned if we do use statistics to help
us do our jobs.
Damned if we don't
Where dozens of social and economic issues
compete every day for media attention, public
awareness, political support and a bigger share of
public funding, nearly every social, political and
economic argument is backed up by statistical
research. In the month of August 2003 alone,
statistical data supported reports in the Toronto Star
and Globe and Mail on baby boomers’ retirement
worries, childhood obesity, crime rates, drug
trafficking, the economic integration of immigrants in
Canada, the effects of family income on university
attendance, gender differences in performance on
standardized reading tests, homelessness, home-
schooling, infant mortality, major depression and
other mental illnesses among Canadians, and seniors
requiring home care services.
Most advocates find it impossible
to move literacy issue
spotlight and onto the public
to literacy rate statistics.
Journalists, decision makers and corporate leaders
with statistics that point to staggering numbers of
Canadians with literacy difficulties. Most advocates
find it impossible to move literacy issues into the
spotlight and onto the public policy agenda without
referring to literacy rate statistics. And the influence
of literacy rate statistics doesn’t end there; it is seen
throughout the policy development process.
Once decision makers have bought into
the notion that literacy warrants a public policy
response, the issue literacy has to be framed further.
What is the specific problem in need of attention?
Who suffers from it and how? Who gains from it?
What causes it? Population literacy surveys, statistics
that derive from them, and interpretations of that
data all influence the way literacy is framed as a
policy issue. For example, reports on the
International Adult Literacy Survey (IALS), issued by
the OECD and Statistics Canada, frame literacy as a
human capital issue, crucial to the economic
performance of industrialized nations in an
increasingly competitive global economy.
Forecasting is used in policy development
to help decision makers make better decisions. Where literacy
is regarded as a public policy concern, literacy rate
data is bound to show up in the forecasts. For
example, consider the following notes from a
presentation on the federal government’s Skills and
Learning Agenda (1999).
- Canada’s demographics mean there will be
fewer new workers.
- Shortages of skilled workers could restrain
future growth and innovation.
- 8 million working age Canadians have low
liter- acy skills (IALS levels 1 and 2) by international
- An additional 6.5 million (IALS level 3) will
need continuous upgrading/lifelong learning to
participate effectively in the knowledge-based
Goal-setting and decision-making about the
allocation of public resources are also essential for
policy development. These usually involve a priority-setting process
in which problems, goals, services, geographic areas and/or specific population
are ranked. Data from IALS has been used to make
the case for placing adult literacy at the top of the
priority lists of industrialized nations, primarily as a
labour force development issue. Most Canadian
literacy advocates acknowledge that reports from IALS
have been instrumental in preserving or increasing
funding for literacy.
Once policy goals are determined, options for
valuated and selected. Some argue that literacy assessment data from
large-scale surveys must be used to inform decisions about
how best to address literacy problems.
Without such data it is difficult
to determine what types of educational programs
are needed and where funding should be
channeled. For example, national data
can be used to determine where English
literacy programs and native language
literacy services or bi-lingual literacy
services are needed. (Wiley)
Finally, policy outcomes are reviewed
development process are commonly the
benchmark against which outcomes are
measured. When expectations have been
expressed in terms of changes in literacy
rates, this data becomes a yardstick
used to evaluate policy outcomes. For
example, the 2001 Throne Speech
included the goal of “significantly
increasing the proportion of adults
with higher-level skills.”
In a nutshell, the issue of adult
literacy likely never would have
gained the public and political
recognition it enjoys in Canada
today were it not for the startling
statistical results of national
literacy surveys like IALS. To a
considerable extent, the design
of current policies and
programs has been influenced
by interpretation of literacy
"Statistics are human beings
with the tears wiped off."
The Current Literacy Target
(based on IALS, 1994)
Target When Literacy Rates are
Improved at Some Point in the Future
Damned if we do
Few people, including those Canadians involved in
adult literacy work, have ever formally studied the
sciences of statistics or research design. Thus few can
confidently examine statistical research findings and
interpret them independently. People working in the
literacy field who do have a research background
rarely have the time or mandate to critically consider
new research. As far as many people are concerned,
statistics are alien and alienating.
Meanwhile, most research experts agree that
all existing methods of estimating literacy rates have
significant conceptual and technical limitations. For
starters, there’s no broad consensus about what it
means to be literate. This is a big problem. Logic
dictates that a shared understanding of the word
literacy is a prerequisite for a shared understanding
of how to measure it.
Even those who define
literacy in the same way
may disagree about if
and/or how it should be
measured. Words used in
a definition are one thing;
the phenomenon captured
by a particular assessment
method is another. Many
researchers question the
validity of all assessment
methods used so far, arguing
that they fail to reflect how
real people use literacy skills
in their real lives.
The science of statistics
complicates the problem
further when the results of
thousands of individual literacy
assessments are aggregated into
population literacy rate estimates.
Population statistics tend to
diminish the complex realities of
individuals in service of creating mathematical
findings. This problem led one pundit to say, “Statistics are human beings
with the tears wiped off.”
Different experts can legitimate ly draw different
conclusions from the same data. A skilled
mathematician who tortures numbers long enough
can make them confess to almost any thing. The
pliable nature of statistics led Mark Twain to say, “There are three kinds of lies – lies, damned lies and
statistics.” Many of our colleagues are wary of
statistics believing that numbers can and will be
manipulated to prove whatever a researcher wants
them to show.
Numbers that quantify Canada’s literacy challenges may engage us, but they may
also enrage us. They may reveal a lot, but they may conceal as much.
Thus literacy advocates and policy makers are damned if we do and damned if
we don’t use
literacy statistics to help us do our jobs. Without
them we seem unable to gain and keep public and
political support for broad-based strategies to
improve literacy levels. While the statistics can help
put the issue on the map, they don’t always lead us
in the right direction. Thus numbers that quantify
Canada’s literacy challenges may engage us, but they
may also enrage us. They may reveal a lot, but they
may conceal as much. While discourse on literacy is
influenced by the numbers at the policy level,
literacy rate estimates don’t necessarily reflect
literacy problems as learners and practitioners in
programs understand them.
We need to understand statistics, and
when, how and why they are used.
Tensions between the opportunities and challenges
associated with using literacy statistics generate noise and confusion in
system of literacy initiatives. What get said about literacy in public awareness
campaigns (e.g. “22 per cent of adult
Canadians have serious difficulty dealing with print”)
may be highly relevant to literacy rate statistics but
not entirely relevant to what goes on in literacy
programs, which tend to be shaped more by realities
and needs of individual learners and practitioners.
Many of the important gains made in literacy
programs may never show up in literacy rate statistics.
Thus we run the risk of winning public and political
support today because of what the numbers show,
only to loose it tomorrow because of what the
numbers won’t show.
What to do?
For better or worse, literacy rate statistics will
continue to be used wherever literacy policy decisions
Canada’s literacy policies cannot simply ignore the
tead, we need to understand them and
when, how and why they are used. We don’t
necessarily need to become statisticians but we do
need to know what questions to ask about large scale
literacy assessment research, and we need to develop the will and the
ability to become critical consumers of literacy rate statistics. Towards
these ends, I offer four recommendations.
- Given the potential influence of literacy rate
data on the development of policies and
programs, alternative plausible interpretations
of literacy rate data should always be identified
and considered before that data is used to
inform policy decisions.
- Researchers from various professional affiliations and
theoretical persuasions should be involved in interpreting literacy
rate data as
early as possible in data-analysis and policy-
- Consumers of research should be encouraged
and supported to reflect critically on literacy
rate data and related interpretations.
- Training for advocates, policy-makers
and other interested parties in how to critically review
literacy rate research should be included in
initiatives to build Canada’s literacy-related
Susan Sussman has worked in literacy since 1993, in the position of Executive Director of the Ontario Literacy
Coalition, as the President and board member of the Movement for
Canadian Literacy, and as an independent researcher. Sussman holds a
master’s degree from Columbia University's Teachers College and a
bachelor's degree from the City University of New York.
Skills and Learning Agenda 1999. Notes from a presentation by a representative
from Human Resources Development Canada, at a meeting of invited literacy
stakeholders convened by Senator Joyce Fairbairn, to gather input on
proposed Skills and Learning Agenda. Ottawa.
Speech from the Throne to
Open the First Session of the Thirty-Seventh Parliament of Canada,
Ottawa, January 30, 2001. Her Excellency the Right Honourable
Wiley, Terrence G. (1994) “Estimating Literacy in the Multicultural
Issues and Concerns”. Washington D.C.: ERIC Digests.