Evaluation of the Electoral Reminder Program (ERP) for the 42nd Canadian Federal Election
Introduction
Nielsen is pleased to present this report to Elections Canada highlighting the findings from the evaluation of the Electoral Reminder Program (ERP) for the 42nd Canadian federal election surveys. It is based on quantitative and qualitative research conducted to better understand the perceptions and awareness of Canadians of Elections Canada's ERP.
Objectives
The ERP was an important program for Elections Canada. It targeted many audiences and was aimed at providing the key information to eligible voters in preparation for the 42nd Canadian general election. Specifically, its goal was to inform Canadians about when, where and the ways to register and vote. As this was a significant program undertaken, Elections Canada needed to evaluate the ERP. Specifically, the objectives of the evaluation were to understand:
- Program rationale and relevance:
- Is the ERP responding to stakeholders' information needs?
- Program design and implementation:Footnote 2
- To what extent are ERP products and/or activities reaching intended audiences?
- Program impact:
- To what extent are intended immediate outcomes being achieved?
- Recall
- Comprehension
- Awareness
- Call to action
- Accessibility and/or usability of website
- To what extent are intended intermediate outcomes being achieved?
- Knowledge
- Elections Canada as primary source of election information
- To what extent are ultimate outcomes being achieved?
- Actual experience
- To what extent are intended immediate outcomes being achieved?
Methodology
Quantitative Surveys
A five-phase quantitative online panel survey, as well as a quantitative post-campaign telephone survey, was used to gauge recall of Elections Canada communications materials and the overall effectiveness of key aspects of Elections Canada's communications campaign by gauging Canadians' knowledge, understanding and perceptions of the voting process. The quantitative program began with a benchmark online study prior to the launch of the campaign, followed by several waves of online tracking as each phase of the ERP was rolled out, ending with an online and telephone post-campaign review. Each phase of the communications campaign was targeted to specific audiences, and this was reflected in the surveying that was completed at each phase.
The survey also provided an opportunity to test phone vs. online-mode completion in the post-campaign evaluation; those findings are presented under separate cover.
Respondents for this survey program were selected from among those who had registered to participate in online surveys. The results of such surveys cannot be described as statistically projectable to the target population. The data have been weighted to reflect the demographic composition of the Canadian population. Because the sample is based on those who initially self-selected for participation in the panel, no estimates of sampling error can be calculated.
The following table summarizes the ERP focus, dates and target audiences in each survey phase.
Phase: Campaign Period | Survey Fielding Dates | Target Audience |
---|---|---|
Phase 1: Benchmark survey | August 10–20, 2015 | General public, youth, students, ethnocultural and Aboriginal electors, those with disabilities |
Phase 2: Registration campaign | September 21–October 4, 2015 | Young Canadians, new citizens, recent movers, ethnocultural and Aboriginal electors |
Phase 3: VIC/Advance polls/Reminder registration campaign | October 9–18, 2015 | General public, youth, students, ethnocultural and Aboriginal electors, those with disabilities |
Phase 4: Election Day campaign | October 20–November 1, 2015 | General public, youth, students, ethnocultural and Aboriginal electors, those with disabilities |
Phase 5: Post-campaign survey | October 20–November 1, 2015 | General public, youth, students, ethnocultural and Aboriginal electors, those with disabilities |
The following table presents the number of surveys completed in each phase by target audience.
Target Group | Phase 1: Benchmark | Phase 2: Registration | Phase 3: VIC/AP/RB | Phase 4: Election Day | Phase 5: Post-campaign |
---|---|---|---|---|---|
Total | 1,256 | 1,519 | 1,112 | 1,046 | 1,072 |
Students | 100 | 518 | 104 | 101 | 105 |
Aboriginal electors | 104 | 109Footnote 3 | 291Footnote 3 | 122Footnote 3 | 95 |
New voters | 95 | 500 | 101 | 118 | 95 |
Disabled electors | 108 | – | 93 | 94 | 88 |
General population | 849 | 501 | 814 | 733 | 689 |
Ethnocultural electorsFootnote 4 | 106 | – | 111 | 162 | 170 |
The questionnaire used to complete the survey was based on the Advertising Campaign Evaluation Tool (ACET), a standard tool used by the government of Canada that tests awareness, recall and comprehension of advertisements. Additional sections were included to gauge electors' awareness and knowledge of responsibilities and actions. The bilingual survey took an average of 15 minutes to complete in each phase. The survey data were weighted to reflect the Canadian population by target group, region, age and gender.
The following table presents the advertisements and communications pieces tested in each phase.
Phase | Phase 2 September 21–October 4, 2015 |
Phase 3 October 9 - 18, 2015 |
Phase 4 October 20–November 1, 2015 |
||
---|---|---|---|---|---|
Campaign/Medium | Registration | VIC | Advance Polls | Communications Pieces | Election Day |
Out of home: public transit/campus | ✓ (X ballot) ✓ (birthday cake) |
✓ | ✓ | – | – |
Out of home: Tim Hortons | ✓ (X ballot) ✓ (birthday cake) |
✓ | ✓ | – | – |
Print ad | ✓ | ✓ | ✓ | – | ✓ |
Radio ad | ✓ | ✓ | ✓ | – | ✓ |
TV ad | ✓ | ✓ | ✓ | – | ✓ |
Internet ad | ✓ | ✓ | ✓ | – | ✓ |
Facebook ad | ✓ | ✓ | ✓ | – | ✓ |
Twitter ad | ✓ | – | – | – | ✓ |
Student postcard | ✓ | – | – | – | – |
Voter information card | – | – | – | ✓ | – |
Reminder brochure | – | – | – | ✓ | – |
ID tear-off sheet/poster | – | – | – | ✓ | ✓ |
Qualitative Focus Groups
The primary focus of the qualitative phase was to conduct an in-depth evaluation of the effectiveness of the communications campaign. The ERP advertising and communications pieces (i.e. direct-mail campaign and direct marketing products) were tested to better understand the successful and less successful aspects of the campaign. Seventeen in-person and online focus groups were conducted across the country with Elections Canada's target groups: youth 18–24, the general population 25+, Aboriginal electors, ethnocultural electors and those with disabilities.
The following grid outlines the cities and target audience of each session.
Table 4 – Summary of ERP evaluation focus groups
General Population (25+) |
General Population (18–24) |
Aboriginal Electors | Ethno-cultural Electors | Disabled Electors | Total | |
---|---|---|---|---|---|---|
Halifax, NS | 1 | 1 | 2 | |||
Montreal, QC | 1 (French) | 1 (French) | 1 (English) | 3 | ||
Mississauga, ON | 1 | 1 | 1 | 3 | ||
Winnipeg, MB | 2 | 2 | ||||
Vancouver, BC | 1 | 1 | 1 | 3 | ||
Online | 1 | 3 | 4 | |||
Total | 3 | 5 | 3 | 3 | 3 | 17 |
The sessions were standard in-person or online focus groups of two hours in length. Ten people were recruited and confirmed, aiming for between eight and ten participants. They each received an honorarium of $75 in appreciation for their time. The sessions were conducted in the evening.
In Reading This Report
This report includes an executive summary and a detailed analysis of the tracking results. Appended to this report is the questionnaire used to complete the survey, the recruitment screener used to invite participants to the focus groups and the discussion guide used by the moderator during the focus groups.
Readers should keep in mind the following in reading this report:
- Percentages may not add up to 100% due to rounding.
- Closed-ended questions (e.g. level of feeling informed) were asked using a 0-to-10 scale, with 10 being the best possible score, 0 being the worst possible score and 5 being neutral.
- The percentages shown for the highest response categories (sometimes referred to as "top 3 box" or "at least somewhat familiar") were calculated by adding the raw scores and dividing by the total. As a result, they may be different than simply adding the percentages from the graphs.
- To avoid leading respondents to the "correct answers," many of the questions were open ended. As such, the intensity of the results is naturally lower than if respondents had been asked to select answers from lists provided.
- Results of unaided recall may include ads and/or communications pieces beyond Elections Canada. For example, responses such as "direct mail" include material beyond the voter information card (VIC), reminder brochure and possibly registration letters. Results of aided recall are based on a written description of the ad or communications piece.
- In some graphs, the survey question text may be shortened to convey the meaning of the question while maintaining a readable font size. The full questionnaire distributed to respondents may be found in Appendix A.
- Statistical significance testing has been applied to determine whether, statistically speaking, differences exist among comparison groups or whether two numbers can be considered statistically the same.
- Where applicable, responses that differ between survey phases are noted. An arrow () or an asterisk (*) indicates that a measure has significantly changed at the 95% confidence interval. As a result, we are 95% certain that there has been a change between phases and it is not due to chance.
Return to source of Footnote 2 This section of the evaluation was measured using tools and data sources other than the public opinion research conducted by Nielsen.
Return to source of Footnote 3 Phases 2, 3 and 4 of the Aboriginal survey used a partner to drive traffic to the online survey; as such, these completions are not combined with the core samples for these phases.
Return to source of Footnote 4 For the purposes of this evaluation, ethnocultural electors are defined as those whose mother tongue is neither French nor English. They are a subset of the specific groups that were targeted for quotas and as such should not be added to other target groups to determine the total sample size.