Annex 1 Evaluation Matrix

Evaluation Questions

Indicators

Methods/Sources

Q1. To what extent is there a continued need for R&D, production, monitoring and service delivery in support of weather prediction?

• Evidence of continued need

• Document review

Q2. To what extent is R&D, production, monitoring and service delivery in support of weather prediction consistent and aligned with federal government priorities and departmental strategic outcomes?

• Evidence of alignment with federal government priorities
• Evidence of alignment with departmental strategic outcomes

• Document review

Q3. Is there a legitimate and necessary role for government to undertake scientific activity in support of weather prediction?

• Evidence of consistency with federal roles and responsibilities

• Document review

Q4. To what extent and in which circumstances are the needs of internal and external clients of R&D, production, monitoring and service delivery in support of weather prediction being met?

• Stated and documented needs and expectations of various weather prediction clients
• Satisfaction of interviewed weather prediction clients with extent to which their needs are currently being met
• Examples of circumstances in which needs are met or not met

• Document review
• Interviews

Q5. In what situations, contexts and circumstances does the current approach to knowledge transfer used by weather prediction work, and for whom?

• Evidence of science knowledge transfer practices
• Key informant opinions on the adequacy and effectiveness of existing science knowledge transfer

• Document review
• Interviews
• Review of literature

Q6. To what extent and in what circumstances are priority‑setting and decision‑making mechanisms and practices pertaining to R&D, production, monitoring and service delivery in support of weather prediction effective for meeting weather prediction client needs?

• Evidence of priority‑setting and decision‑making mechanisms being in place
• Key informant opinions on the adequacy and effectiveness of existing priority‑setting and decision‑making mechanisms

• Document review
• Interviews

Q7. In what situations and circumstances does the QMS appear to be working, and for whom? Why?

• Evidence of positive effects of the introduction of QMS
• Opinions of key informants on the benefits of QMS

• Document review
• Interviews

Q8. What external factors influence the capacity of R&D, monitoring, production and service delivery to support weather prediction, and how?

• External factors

• Document review
• Interviews

Q9. Does the level of effort expended for weather prediction activities yield commensurate value? Are there more efficient ways of delivering science to support weather prediction?

• Analysis of costs and achievement of outcomes
• Views of key informants

• Document review
• Interviews