Follow-up Audit of P3 Canada Fund Screening Project Ranking Process

Follow up Audit Summary: Follow-up Audit of P3 Canada Fund Screening Project Ranking Process (Report dated March 2015)

The objective of this follow-up audit was to assess the implementation of the approved management action plan for the audit of the P3 Canada Fund screening and project ranking process, and to ensure that the steps taken and corrective actions put in place by management are mitigating the risks identified in the original audit.

The scope of this audit engagement is from the Request for Proposals (RFP) for Round Six of the P3 Canada Fund project applications, which were received by the submission deadline of June 13, 2014, until the submission of recommended applications to the Board of Directors. 

Overall, based on the scope above, we found that management has addressed all of the recommendations provided in the April 2013 audit of the P3 Canada Fund screening and project ranking process.

The following table presents specific findings from the audit of the P3 Canada Fund screening and project ranking process reported in April 2013; it provides a description of the original audit recommendation, management action plan, and the current observation and conclusion.


Original Audit Recommendation

 Management Response and Action Plan

Follow-up Observations and Conclusions

We recommend that management develop a practice to formally evidence participation and agreement with the final Evaluation Group assessment for audit trail purposes. PPP Canada agrees with the audit recommendation that formal practices to evidence participation and agreement with final Evaluation Group assessments should be developed.

The Evaluation Group assigned to review and assess P3 Canada Fund applications is made up of skilled and experienced staff from the Project Development and Investments business units. Group members have multiple years’ experience evaluating and supporting P3 projects. 

PPP Canada will develop a formal participation list for the Evaluation Group responsible for project screening and ranking. Once the group has reached consensus on project scores, all participants will be required to sign a routing slip as evidence of their agreement with the final scores. Final consensus documentation will be saved on P3C Docs in the respective folder for audit trail purposes.

√ - Recommendation adequately addressed

Management has developed a practice to formally evidence participation and agreement of final scores from the Evaluation Group.

Final project scores entered into the Project Information Sheets (PIS) are replicated in the database sheet and aligned with the routing slip signed by the Evaluation Group.

We recommend that management develop a practice to evidence approval of Other Levels of Government (OLG) Committee decisions and include additional rationale for key OLG decisions regarding project ranking cut offs and strategic project considerations.

PPP Canada agrees with the audit recommendation. 

PPP Canada has made significant progress in developing formal, systematic processes and internal controls to manage and support internal governance arrangements. More recently, PPP Canada has developed documented committee terms of reference, meeting minutes and decision points for each governance committee. The corporate secretary is responsible for secretariat support and for distributing the minutes and decision points to the management team for review. The management team has the opportunity to challenge and comment on the minutes and decision points at subsequent meetings. Once consensus has been reached on the accuracy of the minutes and decision points, they are approved by the Chair and saved on P3C Docs in the respective folder for audit trail purposes.

Moving forward, the Evaluation Group will prepare documentation surrounding its rationale and decision associated with project ranking cut-offs. Furthermore, the OLG Committee will document its rationale and decision associated with the selection of strategic projects. The rationale for both the cut-offs and strategic projects will be presented to the OLG Committee for consideration and approval. Supporting documentation will be saved on P3C Docs in the respective folder for audit trail purposes.

√ - Recommendation adequately addressed

Appropriate information is provided to the members of the OLG Committee for decision-making purposes during the P3 Canada Fund project application screening process. The Evaluation Group provides the OLG Committee with their P3 Canada Fund screening and project ranking. Rationale and decisions associated with the selection of both the cut-offs and strategic projects are presented to the OLG Committee for consideration and approval.

After the OLG Committee reviews the projects proposed by the Evaluation Group, decisions regarding project ranking cut offs and strategic project considerations are not documented in separate decision points. Rather, results of decisions taken by the OLG Committee, which include the project ranking and rationale for screened-out projects, are summarized in a formal submission presented to the Board for its consideration and recommendation to the Minister.

We recommend that management include a review of the application submission date as part of the qualitative assessment phase prior to assessing application merit, and all applications received after the application deadline should be disqualified from the screening process.

PPP Canada agrees with the audit recommendation. 

PPP Canada conducts a detailed qualitative review on the completeness of all P3 Canada Fund applications prior to evaluation and scoring. Moving forward, all applications, including those from regional contacts, interlocutors and representatives from other levels of government, will be required to submit their files to a dedicated email address (PPPadministration@p3canada.ca) on the date and time prescribed in the Application Guide. Applications received after the prescribed deadline will be automatically disqualified from the screening process. 

The Project Development business unit will institute a formal checklist procedure to ensure that mandatory criteria associated with the screening process are met.

√ - Recommendation adequately addressed

P3 Canada Fund Round Six Application Guide includes a clear deadline.

The P3 Canada Fund Application Evaluation/Assessor’s Guide assesses application submission deadline and mandatory criteria associated with screening process. The guide’s checklist is replicated in the Project Information Sheet (PIS). 

P3 Canada Fund project applications reviewed were submitted to the designated email address and prior to the established deadline.

It is recommended that management segregate duties associated with the project information and introduce a formal change management process to manage all changes, updates or modifications in the Project Information Sheet (PIS).
PPP Canada agrees with the audit recommendation. 

Over the 2012–13 planning period, PPP Canada led the implementation of an Electronic Document and Record Management System (EDRMS). The system is designed to enhance PPP Canada’s capability to manage documents and records, and strengthen information security. By implementing the EDRMS, PPP Canada has the ability to manage access rights controls and keep a clear audit trail on any modifications and deletions to document versions and their associated profiles. Moving forward, PPP Canada will manage the P3 Canada Fund PIS within the EDRMS. The Project Development business unit will grant a select few resources with editing access rights to the PIS. Any changes made to the PIS will be managed by these resources and evidenced by an audit trail within the EDRMS.

√ - Recommendation adequately addressed

As noted in the introduction, the screening and project ranking process has changed slightly from the time of the original audit. The database sheet is now used to consolidate all of the individual PIS and is programmed to link to individual PIS. The database sheet, developed in MS Excel, is considered a data aggregation tool, used to simplify the process of pulling together information contained in PIS. 

Prior to the start of Round Six, management reported that PPP Canada followed a systematic process to validate the suitability of the database sheet design. Before entering into the production environment, multiple mock project information sheets were reportedly prepared and tested in the project database in a separate test environment. The Project Development business unit approved the accuracy and performance of the database sheet, prior to it being put into production. While Round Six was active, the production environment was controlled by password, restricting any user from making any adjustments to the macros (VBA) without proper authorization.

The follow-up audit found that with regards to the database sheet, the Project Development associate does have the ability to modify data fields in the database sheet and could manipulate data which is reported to senior management and used for decision-making purposes.

That said, the audit noted that there are subsequent detective mitigating controls which reasonably could identify unapproved changes to project information. Specifically, the information extracted from the database sheet is reviewed by both the director and VP, Project Development, prior to submitting the recommended screening and project ranking to the members of the Other Levels of Government (OLG) Committee. The number of screened projects has diminished significantly from previous rounds (12 in Round 6), further reducing the likelihood that any changes would be undetected. Lastly, there are detective audit trails created in the EDRMS (P3C Docs) which would serve as deterrence and further reduce this risk.

It is recommended that management conduct a review of the scoring approach and methodology for future rounds to ensure that clearly defined evaluation metrics are used which are consistent with the documented criteria and scoring grid. PPP Canada agrees with the audit recommendation.

PPP Canada continuously monitors and reviews the processes, procedures and approach used to execute each round of the P3 Canada Fund. Lessons learned and experienced gained from previous rounds are used to inform the development of processes and procedures for future rounds. 

Work is currently underway to update and strengthen the P3 Canada Fund Application Evaluation/Assessor’s Guide to enhance consistency with the criteria and scoring grid. PPP Canada will continue to provide staff with application assessment training.

Supporting documentation, final project scores and their respective ranking will be updated on the Electronic Document and Record Management System (EDRMS) in the respective folder for audit trail purposes.

√ - Recommendation adequately addressed

P3 Canada Fund Round Six Application Evaluation/Assessor’s Guide includes clear evaluation metrics.

The Evaluation Group final consensus project score, as presented to the members of the Other Levels of Government (OLG) Committee, is aligned with the project scores as recorded in the EDRMS Project Information Sheets (PIS).