Guidelines for Implementing the PRIA Measurement and Evaluation Model and Matrix

The Public Relations Institute of Australia (PRIA) Evaluation Framework for strategic communication, incorporating the PRIA Evaluation Model and the PRIA Evaluation Implementation Matrix, should be interpreted and applied noting the following guidelines.

  1. All communication program or campaign[i] planning should start with SMART communication objectives – i.e., objectives that are specific, measurable, achievable, relevant, and time-bound. SMART objectives are characterised by the following:
  • Specific – communication objectives should contain numbers such as target volumes or percentages and dates (e.g., ‘to increase target public awareness of sugar as a key cause of obesity to 80% or more by 30 June’ or ‘to gain 10,000 registrations by 1 December’);
  • Measurable – three key features that make communication objectives measurable are (a) they should be specific as noted above; (b) baseline data should be available for post-program comparison (see Note 3); and (c) evaluation should be planned as part of strategic communication planning to ensure that necessary data collection is incorporated and budget is set aside;
  • Achievable – formative research such as review of relevant literature and case studies (see Note 9) can identify whether the proposed communication objectives are realistic (e.g., have similar programs elsewhere achieved the intended results);
  • Relevant – communication objectives must align with and support overarching organisation objectives. In addition, communication objectives should take account of the needs and interests of stakeholders and publics (see Note 2);
  • Time-bound – the deadline for achievement of objectives should be stated.
  1. Communication objectives should directly support one or more organisation objectives. In addition, social responsibility (CR/CSR) requires that communication objectives also should be developed with consideration of the needs and interests of stakeholders, publics, and society in general. Communication objectives that serve the needs of an organisation, but intentionally or knowingly disadvantage some stakeholders, publics, or sectors of society (e.g., social welfare or the environment) are not socially responsible. This holistic approach to setting communication objectives is illustrated by the bidirectional arrows in the PRIA Evaluation Model. Also, the setting of communication objectives is shaped by formative research that provides insights into target publics’ attitudes, concerns, interests, etc. – hence the setting of SMART objectives overlaps with the ‘inputs’ stage of communication (see Note 4 in relation to stages).

  1. Baseline data is important for evaluation of outcomes and impact (e.g., through pre- and post-program comparative analysis). For example, in the first sample objective above, evaluation of increased target public awareness will require data on pre-campaign levels of awareness. Collection of baseline data should be part of formative evaluation (i.e., ex-ante) conducted during the ‘inputs’ stage (see Note 4).

  1. The identification of strategic communication in stages as inputs, activities, outputs, outcomes, and impact is based on program logic models that are widely-used in program planning and evaluation within public administration, education, and international development. For example:
  1. These types of models can be used for planning and evaluation by government, corporations, non-government organisations (NGOs), and non-profit organisations and they can be applied to all types of strategic communication including paid media advertising, media publicity, events, publications, Web sites, stakeholder engagement, and internal organisational communication, as well as integrated campaigns – although different methods are used for evaluating different types of objectives at various stages. (See Note 14 and the accompanying Evaluation Implementation Matrix.)

  1. Some program logic models for communication use slightly different terms for the key stages. For example, some include ‘outtakes’, ‘outgrowths’ or ‘outflows’ as additional or alternative terms for stages. The International Association for Measurement and Evaluation of Communication (AMEC) uses six-stages in the AMEC Integrated Evaluation Framework (inputs, activities, outputs, outtakes, outcomes, and impact)[v]. ‘Outtakes’ is a term used in some public relations literature to refer to short-term interim outcomes in the process of communication such as audience attention and awareness of messages. PRIA supports and endorses the AMEC Integrated Evaluation Framework and recommends the AMEC interactive online application for reporting evaluation. However, what some refer to as ‘outtakes’ are incorporated within ‘short-term outcomes’ in the PRIA Evaluation Framework for simplicity and because an international study of evaluation models has found that the most common descriptions of program stages are inputs, activities, outputs, outcomes, and impact[vi]. (It should be noted that the interactive online AMEC Integrated Evaluation Framework does not require users to enter data in all sections/modules. Users may decide to describe short-term outcomes as ‘outtakes’ or include these under ‘outcomes’.) See the PRIA Evaluation Implementation Matrix that explains the alignment between ‘outtakes’ and ‘short term outcomes’.

  1. The stages of inputs, activities, outputs, outcomes, and impact, while often shown as separate elements for ease of illustration, are not discrete stages in a simple linear process. In reality, the stages overlap and accordingly are illustrated that way in the PRIA Evaluation Model. For example, ‘activities’ continue during the stages of ‘outcomes’ and ‘impact’ to maintain effects. Also feedback and learning gained at each stage should be used to refine strategy and adjust tactics if necessary (see Note 8.)

  1. Feedback loops’ are vital elements of the model. For example, if audiences are not responding to an output such as an information Web site, it may be necessary to adjust strategy or even return to redesign some activities and outputs, or even return to collect additional inputs (e.g., more target public insights) and sometimes even revise communication objectives (e.g., if there is a change in context). (See arrows denoting feedback loops above all stages in the PRIA Evaluation Model.) In some cases, it may even be necessary to review and revise communication objectives based on feedback or changes in context (see Note 14).

  1. The use of program logic models to identify the stages of strategic communication with feedback loops between stages highlights that evaluation should be conducted progressively throughout programs – not only at the end. Best practice recommends three types of evaluation:
  • Formative, also referred to as ex-ante, conducted before programs begin (literature review, identifying baselines, gaining audience insights including channel preferences, and pre-testing are examples of formative evaluation);
  • Process evaluation conducted during programs to track outputs and short-term outcomes;
  • Summative, also referred to as ex-post, conducted after programs to evaluate outcomes and impact.[vii]
  1. The program logic model-based PRIA Evaluation Framework for strategic communication also draws on communication and media theories such as the steps of information processing identified by W. J. McGuire and the communication-persuasion matrix[viii]. The AIDA model (attention, interest, desire, action) and similar models used in advertising is a derivative of McGuire’s steps, although the full list of steps in information processing is much more extensive including exposure, attention, understanding, liking, retention, consideration, acquiring skills or knowledge, attitude change, intention, action/behaviour, and advocacy[ix]. All of these steps in communication are reflected in the PRIA Evaluation Framework.

  1. The PRIA Evaluation Framework also aligns with models in public relations and corporate communication that overview the process of planning and program management including the RACE model of PR planning, which stands for research, action, communication, evaluation[x]; the ROPE model which stands for research, objectives, program/plan, evaluation[xi]; the expanded RAISE model[xii], which advocates research, adaptation, implementation, strategy, evaluation; and the ROSIE model[xiii], which slightly rearranges the stages as research, objectives, strategies, implementation, evaluation.

  1. Stakeholders, publics, and society are not only identified as ‘targets’ for information and/or persuasion in the PRIA Evaluation Framework for strategic communication. The framework illustrates that:
  • Stakeholders, publics, and society should be considered in setting communication objectives as recommended in Note 2 and during the ‘inputs’ stage of programs (e.g., understanding their needs, interests, preferred channels, etc.);
  • Stakeholders, publics, and society will be ‘targets’ and receivers of information during the ‘activities’ and ‘outputs’ stages of programs, which are focussed on production, distribution, and exposure. During these stages, evaluation will necessarily quantify what is distributed to stakeholders, publics, and society (e.g., advertising, media publicity, Web site information, etc.) – hence there is a one-way arrow to stakeholders, publics, and society under ‘outputs’;
  • However, communication ‘activities’ also should include dialogue, listening, collaboration, and relationship building. Evaluation of such activities should include identifying the needs, concerns, and experiences of stakeholders, publics, and relevant sectors of society in relation to the program and its messages. In particular, when evaluating ‘outcomes’, attention should focus on identifying the response of stakeholders, publics, and relevant sectors of society. Hence, a reversed one-way arrow under ‘outcomes’ indicates the importance of inviting and processing feedback and response from stakeholders, publics, and society;
  • Whereas most evaluation frameworks and models highlight evaluation of impact only in terms of achievement of the objectives of the organisation, impact should be evaluated in terms of impact on stakeholders, publics, and society including any unintended impacts (positive or negative), not only intended impacts that serve the objectives and interests of the organisation. This bidirectional flow of impact is represented by a two-way arrow under ‘impact’ in the PRIA Evaluation Framework for strategic communication.
  1. Whereas most evaluation frameworks and models also fail to include context as a major determining variable, the PRIA Evaluation Framework for strategic communication highlights that all stages of communication from setting objectives and planning to impact occur within a dynamic external context (e.g., economic, social, cultural, political, and competitive factors), as well as internal context. Context, represented as a textured ‘backdrop’ of communication programs in the PRIA Evaluation Framework should be considered in:
  • Setting objectives (e.g., are the objectives appropriate, relevant, and achievable in the circumstances); and
  • Conducting evaluation – i.e., program evaluation should include evaluation of context. If the context changes, programs may need to be adjusted and sometimes objectives may need to be revised. For instance, if a major political event such as an election occurs, media and public focus will be diverted. Similarly, an economic recession can change government and public priorities. Evaluation of context should involve presentation of empirical data (i.e., evidence) to show the impact of context, not merely subjective claims.
  1. While the process of planning and evaluating strategic communication and the key concepts and principles involved are overviewed in the model, users should refer to the accompanying Evaluation Implementation Matrix for further detail of key steps, metrics and milestones, and methods of evaluation. Under each of the stages in the model (inputs, activities, outputs, outcomes (short-term to long-term), and impact), the matrix provides:
    a. A brief description/definition of what the stage involves;
    b. The key steps in each stage (such as planning, production, distribution, reception, awareness, engagement, interest, attitude change, etc.);
    c. Examples of things that typically occur at each stage (e.g., placing advertising and writing and issuing media releases are ‘outputs’, while Web page views, downloads, and inquiries are ‘short-term outcomes’ or ‘outtakes’);
    d. Metrics and milestones that can be collected or identified to provide empirical evidence of successful completion of each stage. In some cases, metrics (numbers) can be collected, such as media reach statistics, impressions, online views, etc. The term ‘milestones’ is used because in some cases numbers do not tell the story – e.g., strategic relationships might be demonstrated by signing of a partnership agreement or stakeholders agreeing to work collaboratively on a committee); and
    e. Methods for generating evaluation data, including both formal and informal methods and qualitative as well as quantitative research.

  1. The Evaluation Implementation Matrix is a customised ‘matrix’ to help practitioners implement evaluation for all types of programs and for all levels of budget and time as follows:
  • The horizontal axis – The progressive stages of strategic communication from ‘inputs’ to ‘impact’ are arranged in columns across the matrix from left to right. The aim of evaluation should be to progress as far to the right of the matrix as possible – ideally to evaluating impact;
  • The vertical axis – The metrics, milestones and methods are arranged from simple/basic/low cost at the top of the respective sections in the matrix to sophisticated/advanced methods at the bottom of each section. The aim of evaluation should be to go as deep as possible at each stage to generate the most reliable empirical evidence.
  1. The Evaluation Implementation Matrix includes more than 45 examples of strategic communication inputs, activities, outputs, outcomes, and impacts; 45 examples of metrics and milestones that can demonstrate progress and results at each stage; and 35 examples of research methods (informal and formal; quantitative and qualitative) that can be used for evaluation of strategic communication. (Some methods can be used for evaluating more than one type of activity.) Evaluation of most programs will use only a few of the metrics/milestones and methods listed. The purpose of this matrix is to:
    a. Arrange various metrics/milestones and methods to show what is appropriate to each stage in the process of communication (the columns); and
    b. Show a ‘menu’ of options available in metrics/milestones and methods relevant to each stage from basic to advanced (the levels), with corresponding cost, time, and rigour implications.

  1. Reporting formats for evaluation of strategic communication vary. Some organisations have templates for reports or ‘dashboards’. For organisations that do not have a specified format, the AMEC interactive online tool for recording and reporting evaluation is recommended. See Figure 1, which shows a screen shot of the summary page of a campaign evaluation reported in the AMEC interactive online tool.

Figure 1. Summary of evaluation of a campaign reported in the AMEC interactive online tool.

Notes


[i] The term ‘program’ is used after this point to include communication campaigns, projects, and programs including paid, earned, shared, and owned media, events, and other public communication activities.

[ii] Kellogg Foundation. (2004). Logic Model Development Guide. See http://www.smartgivers.org/uploads/logicmodelguidepdf.pdf (Original work published 1998)

[iii] Henert, E., Jones, L., & Taylor-Power, E. (2003). Enhancing Program Performance with Logic Models. Madison, WI: University of Wisconsin-Extension Program. See http://fyi.uwex.edu/programdevelopment/logic-models

[iv] Hatry, H., Houten, T., Plantz, M., & Greenway, M. (1996). Measuring Program Outcomes: A Practical Approach. Alexandria, VA: United Way of America.

[v] Association for Measurement and Evaluation of Communication (AMEC). (2016). Integrated Evaluation Framework. See http://amecorg.com/amecframework

[vi] Macnamara, J., & Likely, F. (2017). Revisiting the disciplinary home of evaluation: New perspectives to inform PR evaluation standards. Research Journal of the Institute for Public Relations, 2(2), 1–21. http://www.instituteforpr.org/research-journal. Macnamara, J. (2017). Evaluating Public Communication: Exploring New Models, Standards, and Best Practice. London, UK: Routledge.

[vii] Macnamara, J. (2017). Evaluating Public Communication: Exploring New Models, Standards, and Best Practice. London, UK: Routledge.

[viii] McGuire, W. (1985). Attitudes and attitude change. In G. Lindzey & E. Aronson (Eds.), Handbook of Social Psychology, Vol. 2 (3rd ed., pp. 233–346). New York, NY: Random House.

[ix] McGuire, W. (2001). Input and output variables currently promising for constructing persuasive communications. In R. E. Rice & C. K. Atkin (Eds.), Public Communication Campaigns (3rd ed., pp. 22–48). Thousand Oaks, CA: Sage.

[x] Marston, J. (1981). Modern Public Relations. New York, NY: McGraw-Hill.

[xi] Hendrix, J. (1995). Public Relations Cases (3rd ed.). Belmont, CA: Wadsworth.

[xii] Kendall, R. (1997). Public Relations Campaign Strategies: Planning for Implementation (2nd ed.). New York, NY: Addison-Wesley.

[xiii] Crifasi, S. (2000). Everything’s coming up rosie. Public Relations Tactics, 7(9), 14–15. New York, NY: Public Relations Society of America.