open Secondary menu

Evaluation of the Electoral Reminder Program (ERP) for the 42nd Canadian Federal Election

Introduction

Nielsen is pleased to present this report to Elections Canada highlighting the findings from the evaluation of the Electoral Reminder Program (ERP) for the 42nd Canadian federal election surveys. It is based on quantitative and qualitative research conducted to better understand the perceptions and awareness of Canadians of Elections Canada's ERP.

Objectives

The ERP was an important program for Elections Canada. It targeted many audiences and was aimed at providing the key information to eligible voters in preparation for the 42nd Canadian general election. Specifically, its goal was to inform Canadians about when, where and the ways to register and vote. As this was a significant program undertaken, Elections Canada needed to evaluate the ERP. Specifically, the objectives of the evaluation were to understand:

  • Program rationale and relevance:
    • Is the ERP responding to stakeholders' information needs?
  • Program design and implementation:Footnote 2
    • To what extent are ERP products and/or activities reaching intended audiences?
  • Program impact:
    • To what extent are intended immediate outcomes being achieved?
      • Recall
      • Comprehension
      • Awareness
      • Call to action
      • Accessibility and/or usability of website
    • To what extent are intended intermediate outcomes being achieved?
      • Knowledge
      • Elections Canada as primary source of election information
    • To what extent are ultimate outcomes being achieved?
      • Actual experience

Methodology

Quantitative Surveys

A five-phase quantitative online panel survey, as well as a quantitative post-campaign telephone survey, was used to gauge recall of Elections Canada communications materials and the overall effectiveness of key aspects of Elections Canada's communications campaign by gauging Canadians' knowledge, understanding and perceptions of the voting process. The quantitative program began with a benchmark online study prior to the launch of the campaign, followed by several waves of online tracking as each phase of the ERP was rolled out, ending with an online and telephone post-campaign review. Each phase of the communications campaign was targeted to specific audiences, and this was reflected in the surveying that was completed at each phase.

The survey also provided an opportunity to test phone vs. online-mode completion in the post-campaign evaluation; those findings are presented under separate cover.

Respondents for this survey program were selected from among those who had registered to participate in online surveys. The results of such surveys cannot be described as statistically projectable to the target population. The data have been weighted to reflect the demographic composition of the Canadian population. Because the sample is based on those who initially self-selected for participation in the panel, no estimates of sampling error can be calculated.

The following table summarizes the ERP focus, dates and target audiences in each survey phase.

Table 1 – Summary of ERP evaluation survey phases
Phase: Campaign Period Survey Fielding Dates Target Audience
Phase 1: Benchmark survey August 10–20, 2015 General public, youth, students, ethnocultural and Aboriginal electors, those with disabilities
Phase 2: Registration campaign September 21–October 4, 2015 Young Canadians, new citizens, recent movers, ethnocultural and Aboriginal electors
Phase 3: VIC/Advance polls/Reminder registration campaign October 9–18, 2015 General public, youth, students, ethnocultural and Aboriginal electors, those with disabilities
Phase 4: Election Day campaign October 20–November 1, 2015 General public, youth, students, ethnocultural and Aboriginal electors, those with disabilities
Phase 5: Post-campaign survey October 20–November 1, 2015 General public, youth, students, ethnocultural and Aboriginal electors, those with disabilities

The following table presents the number of surveys completed in each phase by target audience.

Table 2 – Summary of ERP evaluation target audiences by survey phase
Target Group Phase 1: Benchmark Phase 2: Registration Phase 3: VIC/AP/RB Phase 4: Election Day Phase 5: Post-campaign
Total 1,256 1,519 1,112 1,046 1,072
Students 100 518 104 101 105
Aboriginal electors 104 109Footnote 3 291Footnote 3 122Footnote 3 95
New voters 95 500 101 118 95
Disabled electors 108 93 94 88
General population 849 501 814 733 689
Ethnocultural electorsFootnote 4 106 111 162 170

The questionnaire used to complete the survey was based on the Advertising Campaign Evaluation Tool (ACET), a standard tool used by the government of Canada that tests awareness, recall and comprehension of advertisements. Additional sections were included to gauge electors' awareness and knowledge of responsibilities and actions. The bilingual survey took an average of 15 minutes to complete in each phase. The survey data were weighted to reflect the Canadian population by target group, region, age and gender.

The following table presents the advertisements and communications pieces tested in each phase.

Table 3 – Summary of ERP evaluation ads and communications pieces tested by phase
Phase Phase 2
September 21–October 4, 2015
Phase 3
October 9 - 18, 2015
Phase 4
October 20–November 1, 2015
Campaign/Medium Registration VIC Advance Polls Communications Pieces Election Day
Out of home: public transit/campus ✓ (X ballot)
✓ (birthday cake)
Out of home: Tim Hortons ✓ (X ballot)
✓ (birthday cake)
Print ad
Radio ad
TV ad
Internet ad
Facebook ad
Twitter ad
Student postcard
Voter information card
Reminder brochure
ID tear-off sheet/poster

Qualitative Focus Groups

The primary focus of the qualitative phase was to conduct an in-depth evaluation of the effectiveness of the communications campaign. The ERP advertising and communications pieces (i.e. direct-mail campaign and direct marketing products) were tested to better understand the successful and less successful aspects of the campaign. Seventeen in-person and online focus groups were conducted across the country with Elections Canada's target groups: youth 18–24, the general population 25+, Aboriginal electors, ethnocultural electors and those with disabilities.

The following grid outlines the cities and target audience of each session.

Table 4 – Summary of ERP evaluation focus groups

General Population
(25+)
General Population
(18–24)
Aboriginal Electors Ethno-cultural Electors Disabled Electors Total
Halifax, NS 1 1 2
Montreal, QC 1 (French) 1 (French) 1 (English) 3
Mississauga, ON 1 1 1 3
Winnipeg, MB 2 2
Vancouver, BC 1 1 1 3
Online 1 3 4
Total 3 5 3 3 3 17

The sessions were standard in-person or online focus groups of two hours in length. Ten people were recruited and confirmed, aiming for between eight and ten participants. They each received an honorarium of $75 in appreciation for their time. The sessions were conducted in the evening.

In Reading This Report

This report includes an executive summary and a detailed analysis of the tracking results. Appended to this report is the questionnaire used to complete the survey, the recruitment screener used to invite participants to the focus groups and the discussion guide used by the moderator during the focus groups.

Readers should keep in mind the following in reading this report:

  • Percentages may not add up to 100% due to rounding.
  • Closed-ended questions (e.g. level of feeling informed) were asked using a 0-to-10 scale, with 10 being the best possible score, 0 being the worst possible score and 5 being neutral.
  • The percentages shown for the highest response categories (sometimes referred to as "top 3 box" or "at least somewhat familiar") were calculated by adding the raw scores and dividing by the total. As a result, they may be different than simply adding the percentages from the graphs.
  • To avoid leading respondents to the "correct answers," many of the questions were open ended. As such, the intensity of the results is naturally lower than if respondents had been asked to select answers from lists provided.
  • Results of unaided recall may include ads and/or communications pieces beyond Elections Canada. For example, responses such as "direct mail" include material beyond the voter information card (VIC), reminder brochure and possibly registration letters. Results of aided recall are based on a written description of the ad or communications piece.
  • In some graphs, the survey question text may be shortened to convey the meaning of the question while maintaining a readable font size. The full questionnaire distributed to respondents may be found in Appendix A.
  • Statistical significance testing has been applied to determine whether, statistically speaking, differences exist among comparison groups or whether two numbers can be considered statistically the same.
  • Where applicable, responses that differ between survey phases are noted. An arrow () or an asterisk (*) indicates that a measure has significantly changed at the 95% confidence interval. As a result, we are 95% certain that there has been a change between phases and it is not due to chance.

Footnote 2 This section of the evaluation was measured using tools and data sources other than the public opinion research conducted by Nielsen.

Footnote 3 Phases 2, 3 and 4 of the Aboriginal survey used a partner to drive traffic to the online survey; as such, these completions are not combined with the core samples for these phases.

Footnote 4 For the purposes of this evaluation, ethnocultural electors are defined as those whose mother tongue is neither French nor English. They are a subset of the specific groups that were targeted for quotas and as such should not be added to other target groups to determine the total sample size.