Archived - Audit of eManifest Project Governance and Reporting

May 2014

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

1.0 Introduction

In March 2005, Canada, the United States (U.S.) and Mexico announced the Security and Prosperity Partnership of North America (SPP), a trilateral commitment to work together to build a safer and more economically dynamic North America. The SPP outlines an agenda for greater cooperation in areas such as security, transportation, the environment and public health. The eManifest project is part of the SPP and was introduced in September 2006, with the purpose of harmonizing trans-border commercial processes between Canada, the United States and Mexico, ultimately enabling the Canada Border Services Agency (CBSA or the Agency) to increase the security and prosperity of Canada.

A major Crown project, eManifest is underway at the CBSA and is a part of the Advance Commercial Information (ACI) initiative. This initiative introduced the requirement to electronically transmit data on cargo, conveyance and crew to the CBSA before shipments arrive at the border, across marine and air modes (2004 and 2006, respectively). Once fully implemented, eManifest will require all members of the trade community such as carriers, freight forwarders, and importers in all transportation modes to electronically transmit ACI.

As per the Policy on the Management of Projects (PoMP), eManifest is a class three project, which means that the project is sustaining, tactical and evolutionary. Based on the CBSA’s Organizational Project Management Capacity rating, it requires independent reporting to the Treasury Board of Canada Secretariat’s (TBS) Chief Information Officer Branch. It is also one of the largest projects undertaken by CBSA to date, and has transformational effects that reach nearly every aspect of the internal organization in addition to the implicated external trade community.

To date, eManifest has introduced the ability for the highway and rail modes, as well as freight forwarders, to submit ACI (in 2010, 2012 and 2013, respectively). The remainder of the project focuses on enabling importers to transmit ACI (originally planned for July 2014); implementing electronic re-manifests, air and marine retrofits; and developing the internal risk assessment and business analytics capabilities to use the advance information for targeting and other purposes (originally planned for December 2014).

2.0 Significance of the Audit

The PoMP was developed to ensure appropriate project management is in place to ensure Government of Canada projects are providing value for money and demonstrating sound stewardship in program delivery. As per the PoMP, Deputy Heads are responsible for ensuring a comprehensive approach to managing projects is integrated across the department that ensures accountability for project outcomes is documented, appropriate controls are in place to minimize risk, key project stakeholders are consulted, and outputs and outcomes are monitored and reported.Footnote 1

Large government Information Technology (IT) projects like eManifest are inherently complex, expensive and risky. They also have had a history of cost overruns and delays, and not delivering what had been planned. As per the PoMP, the Deputy Head is responsible for ensuring that these and all types of projects are managed in a manner that is consistent with their assessed level of complexity and risk.

Management’s ability to ensure appropriate project management, monitor progress with respect to time, scope and costs, and to make informed decisions based on observed progress, relies on the critical project information provided to them (e.g., risk items, resource utilization, critical path delays). In addition, reluctance to report bad news about a troubled project has been recognized as an important contributor to IT project failure. Developing an understanding of the governance and reporting processes and controls in place for eManifest were therefore deemed to be significant to the Agency.Footnote 2

The Audit of eManifest Project Governance and Reporting was conducted to determine whether the management processes, practices and controls in place are adequate to ensure that project governance and reporting is sufficient, accurate, complete, consistent and timely.

3.0 Statement Of Conformance

The audit conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of the Agency quality assurance and improvement program. The audit approach and methodology followed the International Standards for the Professional Practice of Internal Auditing as defined by the Institute of Internal Auditors and the Internal Auditing Standards for the Government of Canada as required by Treasury Board’s Policy on Internal Audit.

4.0 Audit Opinion

A governance framework and key reporting processes for the eManfest project have been defined, documented and communicated in order to enable management to effectively manage, monitor and make decisions regarding the project. However, at the time of the audit, these were not working as defined or as effectively as possible.

This results in a medium-high risk exposure to the Agency.

5.0 Key Findings

It is important to note that over the course of the audit conduct (between September and December 2013), the CBSA implemented a new governance and management structure for all of the Agency’s projects to ensure that one single area of the organization has accountability and oversight for their delivery. This has changed some of the high-level governance for eManifest and has increased project control. The specific changes and controls introduced were not explicitly tested in this audit, as they have not been in place long enough for assurance purposes; however, the changes are expected to provide a more rigorous level of oversight, leading to additional assurance of success.

Prior to these changes, the CBSA had put into place a formalized governance framework for eManifest, and had defined and documented roles, responsibilities, and decision-making accountabilities of key oversight committees and project management. In addition, guidance on eManifest’s reporting process for risk management, issue management, change management and schedule reporting has been defined, documented, up-to-date and communicated widely via the internal wiki for eManifest.

However, although the governance framework has been defined, roles and responsibilities were identified, decision-making accountabilities are in place, and committees do receive information to monitor eManifest’s progress, it is unclear how decisions are being made and actions are being taken and followed-up by key governance committees. In addition, the Agency has not monitored its deliverables as effectively as possible, which has contributed to significant slippage in the schedule of the project.

Furthermore, the full project schedule has not been reflected in project status reports, the indicators on the Dashboard do not always reflect project health and information is regularly reported incorrectly on the dashboards.

Although processes are in place to properly manage and monitor the project, they are not being implemented as effectively as possible. Clearer decision making and monitoring, as well as a better understanding of the project’s exact status on scope, schedule and cost would help to ensure that, going forward, the project is delivered on time and on budget.

6.0 Summary of Recommendations

This audit makes the following four recommendations:

  • Monitor the activities of each of the eManifest project governance committees to ensure sufficient, accurate, complete, consistent and timely information is being received, acted on and monitored at the appropriate level; and
  • Ensure that an independent review of the new corporate project governance structure in place within the Agency is conducted after a reasonable period of time, to confirm it works as intended.
  • Ensure that eManifest is monitored and reported on at the deliverable level and variances are identified and corrected, as required, until project completion.
  • Ensure that all aspects of the scope of the eManifest project are reflected in the key project status reports and monitored until project completion.
  • Ensure that the eManifest Project Executive Dashboard is completed as per Treasury Board and Agency guidance, and is monitored for discrepancies until project completion.

7.0 Management Response

The Information, Science and Technology Branch (ISTB) and the eManifest project agree with the findings of the audit. Overall, the audit highlights that the eManifest project has increased control and is currently making positive steps in moving the project forward.

Beginning in 2013 with the arrival of a new vice-president, the Branch has taken a number of steps to improve project management, including governance and reporting, for all projects at the CBSA. Major projects were transferred from the Programs Branch to ISTB under the oversight of a new associate vice-president with extensive experience as a project executive. To support the new structure, the Agency’s Enterprise Project Management Office has implemented Project Portfolio Management and Project Management Frameworks as well as the portfolio management tool, CA Clarity, to provide additional controls for oversight and monitoring of the Agency’s projects. These fundamentals will be fine-tuned on an ongoing basis to ensure the successful transformation of the Branch. The steps taken to date by the eManifest project will be augmented by these enterprise-level controls in order to improve project governance and reporting.

8.0 Changes to the eManifest Governance Structure

The audit reviewed governance and reporting processes in place for eManifest from August 2012 to the end of the audit conduct in February 2014. Over the course of the audit conduct, significant changes occurred to both the governance and reporting structures. For example, in September 2013, the Major Projects Directorate (MPD), which comprises eManifest along with other commercial projects, was moved from Programs Branch to ISTB, along with two other project directorates. This move enabled the CBSA to put into place a new governance and management structure for all of the Agency’s projects and to ensure that a single area of the organization has accountability and oversight for their delivery. Subsequent to this, the highest level committee for the governance of projects, the Vice-President Major Projects Steering Committee (VPMPSC), which previously reported to the Corporate Management Committee (CMC) before its issues were raised to Executive Committee (EC), had been replaced by the Transformation and Innovation Project Portfolio Committee, which reports directly to the EC. In addition, a dedicated Project Board has been established for eManifest. The Board introduces an integrated project team under a single project manager, who has the overall project delivery accountability.

The new governance structure was finalized in December 2013. A logic model as it relates to eManifest, is provided in Logic Model 1.

Logic Model 1

Logic Model 1

  • 1.0 eManifest Governance Process Flow As Of December 2013
    • 1.1 Executive Committee (EC)
    • 1.2 Transformation, Innovation and Project Portfolio Committee (TIPPC)
      • 1.2.1 Strategic Policy and Program Committee (SPPC)
      • 1.2.2 Corporate Management Committee (CMC)
    • 1.3 Project Advisory Council (PAC)
    • 1.4 CBSA Commercial Sub-Portfolio Projects
    • 1.5 eManifest

Also finalized in December 2013, the Enterprise Project Management Office within ISTB issued the CBSA Project Portfolio Management Framework and the CBSA Project Management Framework (PMF), which captures the CBSA’s corporate approach to managing projects that includes standardized processes and a corporate governance structure. The frameworks establish how projects are governed throughout the project management lifecycle, including a gating process from the approved investment to project closeout. It also describes the approval authorities, oversight and incorporates the TBS PoMP, which emphasizes decision making, accountability, communication, consultation and involvement of stakeholders. All projects at the CBSA are now required to follow the framework. Interviewees noted that the new management, governance and processes have increased project control and are expected to provide a more rigorous level of oversight, leading to additional assurance of success.

Although the new governance structure and reporting processes were put into place during the course of the audit conduct, the specific changes and controls introduced were not explicitly tested, as they have not been in place long enough for assurance purposes, nor were they the focus of the audit’s scope. The audit focused on the governance and reporting processes in place until December 2013.

9.0 Audit Findings

9.1 Governance and Reporting Frameworks

Audit Criteria:

  • Effective oversight bodies are established and defined.
  • Management has defined project status reporting requirements that address project schedule, scope, costs, deliverables, issues and risks, and reporting frequency.
  • Risk and issue tolerance threshholds are defined, approved and communicated.

9.1.1 The governance framework has been defined and communicated for the eManifest project.

Prior to the changes noted above, the CBSA had established a governance framework for eManifest, and had defined and documented roles, responsibilities, and decision-making accountabilities of key oversight committees and project management. On September 6, 2006, the eManifest Project Charter was issued; it identified a defined “Project Organization,” which documented the initial roles and responsibilities of the eManifest project team, such as the project manager, the project sponsors, the project team and various other stakeholders. The Charter also included a description of the project governance, which listed the various committees and decision-making bodies that were created to steer the project and ensure that key decisions on eManifest were consistent with the  CBSA’s corporate vision, mission and values. Since the creation of the project, the governance framework and the internal reporting lines have changed, with the MPD moving from ISTB to Programs Branch and then, in September 2013, back to ISTB. These changes have been re-documented via various internal governance documents.

Each oversight committee has documented terms of reference that stipulate each committee’s mandate, members, roles and responsibilities, meeting frequency, reporting and escalation responsibilities, and terms for establishing quorum; these can be found directly within the Internal Governance Structure of the MPD. The decision-making accountabilities, which include document approval authorities and escalation procedures, are also documented and defined within the Internal Governance Structure of the MPD.

The MPD’s Internal Governance Structure also has a clearly defined escalation process that provides a model for escalation and resolution of issues at the various governance committees. The purpose of the process is to obtain decisions and approvals from the necessary governance bodies in a timely fashion during the lifecycle of eManifest. Furthermore, although not formally defined, a quality review and challenge process is in place for the information reported on the project to governance committees. The challenge is done at lower (e.g. operational) and at higher (e.g. director general/vice-president) levels of authority.

The logic model of the governance structure for eManifest prior to December 2013 is provided below in Logic Model 2.

Logic Model 2

Logic Model 2

  • 1.0 eManifest Governance Process Flow Prior To December 2013
    • 1.1 Executive Committee (EC)
    • 1.2 Corporate Management Committee (CMC)
    • 1.3 Vice President’s Major Projects Steering Committee (VPMPSC)
      • 1.3.1 Project Management Stream
        • 1.3.1.1 Directors General Project Steering Committee
        • 1.3.1.2 Commercial Project Directors Steering Committee (CPDSC)
          • 1.3.1.2.1 Business Managers Committee (BMC)
          • 1.3.1.2.2 Business and IT Managers Committee (B&ITMC)
      • 1.3.2 Policy Stream
        • 1.3.2.1 Directors General Program Alignment Committee (DGPAC)
        • 1.3.2.2 Directors Program Alignment Committee (DPAC)
        • 1.3.2.3 Border Commercial Consultative Committee (BCCC) eManifest Sub-Committee
    • 1.4 Senior Project Advisory Committee (SPAC)

In the governance structure in place prior to December 2013, there were two basic “streams”: the Project Stream and the Policy, Program Alignment and Transition (Policy) Stream. Under the Project Stream, the Business Managers Committee (BMC) and Business & IT Managers Committee (BM&ITC) initiated information related to the project, such as information on cost, schedule and scope, and then escalated, as necessary, through the stream to the VPMPSC. Under the Policy Stream, the Border Commercial Consultative Committee eManifest Sub-Committee (an external consultative, non-decision-making committee) initiated discussion on policy and regulatory alignment issues and then escalated these through that stream up to the VPMPSC, which then reported to the CMC and EC, with the Senior Project Advisory Committee (an external advisory committee) providing oversight when needed.

9.1.2 The reporting and escalation processes, tools and templates have been defined and communicated for eManifest.

eManifest employs a number of tools for monitoring, reporting and decision-making purposes. These tools include:

  • eManifest Executive Project Dashboard (the Dashboard);
  • Schedule Status Reports;
  • Master schedule;
  • eManifest Project Benefits Report
  • Implementation snapshots;
  • Risk and Issue logs;
  • Change logs;
  • Ad hoc Reports; and
  • Briefing Notes.Footnote 3

Although the PMF created by EPMO will guide project reporting going forward, guidance on eManifest’s reporting process for risk management, issue management, change management, using MS Project and schedule reporting has been defined, documented, up-to-date and communicated widely via the internal wiki for eManifest. These documents include the escalation processes, tolerance thresholds and all the required steps, from initiation to closure. In addition, the guidance documents are clearly written and sufficiently detailed for implicated stakeholders to understand how to report risks, issues, changes and schedule variations. Guidance on the Dashboard is also defined, documented, up-to-date and communicated.

The reporting frequency is defined for the project dashboard, the master schedule and the schedule status reports. Although it is not defined for other documents such as the risk, issue and change logs, these documents are updated on an “evergreen” basis. Finally, schedule status reports are created bi-weekly, and are kept on the CBSA wiki.

On top of regular project reporting (e.g., on scope, schedule and cost), a number of steps are being taken to improve benefits management and stakeholder engagement across the project. For example, the project team has developed an eManifest Project Benefits Report outlining benefits based on savings, security and service, a Performance Measurement Framework including key performance indicators (KPIs) with critical success factors and metrics. In addition, the “Implementation Snapshot” identifies the uptake of system capability by internal and external stakeholders (e.g., how many highway carriers are sending advance information through the portal, how many training sessions have been provided to border services officers, how many phone calls the Agency has received with questions regarding eManifest).

9.2 Decision Making

Audit Criteria:

  • Oversight bodies request and receive accurate, complete, consistent, and timely information to monitor eManifest’s progress.

9.2.1 The eManifest decision-making process is not working as described

Although the governance framework has been defined, roles and responsibilities were identified, decision-making accountabilities are in place, and committees receive information to monitor eManifest’s progress, it is unclear how project management decisions (i.e., on scope, cost and schedule) are being made and actions are being taken and followed-up by key governance committees. The audit reviewed the terms of reference (TOR), agendas and records of discussion and decision (RODD) for each committee included in the MPD Internal Governance Structure and to identify what was mandated to be reviewed and approved in each committee compared to what is actually occurring. Some key gaps were noted in the project stream.Footnote 4

The following table (Table 1) identifies the key project documentation and reports that are required to be received, approved or reviewed, as per the committees’ terms of reference in the left-hand column under each committee. It then identifies the information that is actually received, reviewed or approved at each committee in the right-hand column under each committee as per the audit’s review of the RODDs and Agendas.Footnote 5

Table 1

  Governance Committees
  BMC B&ITMC CPDSC DGPSC VPMPSC
    TOR RODD TOR RODD TOR RODD TOR RODD TOR RODD
Key Project Artefacts eManifest Project Dashboard         Approval Review Nothing noted Approval Nothing noted Review
Master Schedule         Approval Review Nothing noted Review Approval Nothing noted
Finance Documents                    
Project Management Tools Risk Documents Review Nothing noted Review Review Approval In Agenda*        
Issue Documentation Review Review Review Review Approval In Agenda* Approval Nothing noted    
Change Documentation Review Approval Review Review Approval In Agenda*     Approval Nothing noted
Business Requirements Nothing noted Approval     Approval Nothing noted        
Schedule Status Report N/A Review N/A Review N/A Review        
Implementation Snapshot N/A Review     N/A Review        
Benefits Realization Plan             N/A Review N/A Review
eManifest-related Policies             Approval Nothing  noted    

The table demonstrates that some committees are not receiving all of the project management documentation they are mandated to review and approve, whereas other committees are approving documents they are not mandated to approve. For example, the Commercial Project Directors Steering Committee (CPDSC) is mandated to review and approve the master integrated schedule and the risk, issue, change and business requirements logs, yet it did not formally approve any of these documents throughout the year. The Director General Project Steering Committee (DGPSC) did not review and approve issue logs or eManifest Policies, as per the requirements of its TOR, but it approved the Dashboard, despite this being the responsibility of the CPDSC.

Furthermore, it is apparent that committees are not meeting as frequently as mandated. For example, the BMC is slated to meet bi-weekly. The beginning of 2013 saw the committee meet almost consistently on this basis, but after April, the frequency of the meetings decreased to once-a-month or less. The CPDSC is also slated to meet bi-weekly, yet it did not meet until April of 2013 and only met 12 times over the course of the year. Finally, the DGPSC was slated to meet monthly, but only met three times in 2013. For the weeks/months that the CPDSC and DGPSC committees did not meet, an email requesting review and comment on the Dashboard prior to it being presented at VPMPSC was prepared by the Project Steering and Control Office (PSCO) of MPD; however the effectiveness of this review and challenge process is unclear. Of all the committees in the eManifest governance structure, only the Business and IT Managers Committee and the VPMPSC were meeting as mandated.

Finally, the review of RODDs identified that although RODDs are kept for every committee meeting throughout the year, the action logs that track action items discussed at committee meetings are not being updated accordingly, suggesting ineffective monitoring of decisions made and implementation actions taken.

The audit also found that eManifest project management regularly uses working groups and committees that are outside of the defined governance structure to review project documents and make decisions. For example, a review of each group’s action logs indicated that project management documentation is reviewed and project decisions are made at the Directorate Management Table and the eManifest Project Health Working Group, the latter having been initiated in November 2013. The table below (Table 2) demonstrates what was reviewed at both committees in the past year.

Table 2

  Working Groups
DMT Health
Key Project Artefacts eManifest Project Dashboard Review  
Master Schedule Review Review
Finance Documents Review  
Project Management Tools Risk Documentation Review Review
Issue Documentation Review Review
Change Documentation   Review
Business Requirements   Review
Schedule Status Report Review  

A review of information going through the Director General’s Office and to the Vice-President’s Office also shows that project management information is being raised to senior management, it is just unclear how this information is being used to monitor, challenge and make decisions on the direction of eManifest.

If committees are not meeting as mandated and reviewing the correct information at the appropriate levels on a regular basis, and actions are not identified and monitored, the Agency is at risk of not identifying and acting on issues in a timely fashion.

Recommendation 1:

The Vice-President, Information, Science and Technology Branch, should monitor the activities of each of the eManifest project governance committees to ensure sufficient, accurate, complete, consistent and timely information is being received, acted on and monitored at the appropriate level.

The Vice-President, Information, Science and Technology Branch, should ensure that an independent review of the new corporate project governance structure in place within the Agency is conducted after a reasonable period of time, to confirm it works as intended.

Management Response

Management Response Completion date

Management agrees with this recommendation. In fact, the Branch identified prior to the writing of this report that the root cause of the problem of the project governance reporting was the lack of quality data that was received in order to allow senior management to make informed fact based decisions. In order to address this problem, the Branch has deployed rigorous Portfolio and Project Frameworks that has formalised a gating process by identifying key deliverables that must be presented at each gate for approval and movement to next phase of the project. In support of fact-based decision making, the Branch has re-baselined the eManifest Project from a scope, schedule and cost perspective and as of April 1st, it will be using earned value analysis and accrual-based accounting to remove the subjectivity of the progress reported in the dashboard. This is a major evolution of our monitoring and reporting process.

Building on this strong base and the improvement of monitoring available data quality, in December 2013, the Agency updated the Vice-President’s Major Projects Steering Committee to the Transformation, Innovation and Project Portfolio (TIPP) Committee. This has increased project management rigour through meaningful fact-based gate reviews and precise guidance on corrective actions. Building on the updated TIPP, the eManifest Project will perform a review of its current governance structure to ensure alignment with the new corporate governance structure. The review will determine that the project’s committees have the appropriate mandate, roles and responsibilities to allow for effective and efficient decision making, monitoring and control. The project will review each committee membership to ensure proper representation.

The Branch fully supports a review of the new corporate governance structure to be completed after a reasonable amount of time in order to ensure the effectiveness of the committees.

September 2014

9.3 Variance Monitoring

Audit Criteria:

  • Variances between actual and plan are analyzed and communicated, for project schedule, scope and costs.
  • Management monitors the actual performance of eManifest against the planned results and adjusts course as needed.

9.3.1 The Agency has not monitored eManifest’s deliverables as effectively as possible

In the fall of 2013, after MPD was moved to report to the Vice-President of ISTB and eManifest was subject to this new governance and reporting structure, an in-depth review of the project was conducted on the request of the Associate Vice-President of ISTB. Based on this review, the Associate Vice-President then requested a re-baseline of the project schedule; he also requested that the new schedule be resource-loaded and activity-based to ensure there was sufficient information to support an estimated timeline to project completion. These activities identified that the December 2014 date for project completion was no longer achievable and the solution was at risk of not meeting project objectives.

Until November 2013, when the project review identified that the project would likely be considerably delayed, the Dashboard had indicated that the health of the project schedule was “green,” that is, variance of less than 10 percent.

Although there is ongoing tracking of financial information on the project, as noted in previous audits and reviews, the project has never used Earned Value Management to measure performance, as was originally required by the eManifest Time Management Plan that was developed in September 19, 2007.Footnote 6 At the same time, the PSCO, who is in charge of updating the master schedule, has been receiving schedule updates from all of the various stakeholders involved in the project that are at milestone level and that do not identify the activities or resources required to complete each milestone. Due to this, as well as the lack of an original baseline, the project has been tracking expenditures at burn-rate, and has not been able to align that burn-rate to deliverable completion.

Without an appropriate project management plan and baseline, and an understanding of the plan’s true status, for a project of this size and complexity, it is difficult to know whether the project is on track. Other indicators may identify whether the project is off-track, but to what degree and in what areas would not be discernible.

This inability to effectively monitor the project against an established baseline has been a challenge over the course of the project, which has contributed to the Agency re-profiling funds for eManifest from fiscal years 2006–2007 to 2011–2012. Related to this, the report commissioned by Comptrollership noted:

…the Agency was occasionally required to reallocate project funding internally to fund operational pressures and address internal program integrity issues…From 2006–2007 to 2011–2012, eManifest’s non-project expenditures totaled $104.6M.Footnote 7

Although insufficient monitoring is not the only reason that the project has re-profiled funds and reallocated funding from the project to other Agency priorities, a better understanding of where and how the project’s funds were being spent would have helped to identify the actual capacity of the project and what was required to ensure delivery on time and on budget.

Recommendation 2:

The Vice-President, Information, Science and Technology Branch, in consultation with the Vice-Presidents of the Comptrollership, Programs, Operations and Human Resources branches, should ensure that eManifest is monitored and reported on at the deliverable level and variances are identified and corrected, as required, until project completion.

Management Response

Management Response Completion date
Management agrees with this recommendation. The project is currently developing a revised integrated project plan, which is detailed to the deliverable level and includes resource allocation. Deliverables will be monitored against an established baseline of the integrated project plan, and monitored until project completion, via the monthly dashboards and through CA Clarity. Variances will be tracked and corrected until project completion. The implementation of the CA Clarity tool will be leveraged to ensure that the appropriate areas receive timely information about the project’s progress – this includes Senior Management, the EPMO and Treasury Board through the monthly dashboard. However, it is not expected that the interoperability of the corporate financial systems and CA Clarity be fully in place prior to the eManifest Project completion. The project will leverage the integrated project schedule and the monthly dashboard, augmented by the use of interim Excel spreadsheets to track costs to more discrete levels as an interim process for tracking earned value. July 2014

9.4 Status Reports

Audit Criteria:

  • Project status reports are subject to management quality review and challenge for completeness, accuracy and consistency.

9.4.1 The indicators on the Dashboard do not always reflect project health

The audit team performed a review of the monthly Dashboards from October 2012 to December 2013 to determine whether they accurately reflected the current state of the project; whether they were complete, accurate, sufficient and timely; and whether variances between actual and planned were communicated to the appropriate oversight bodies.

The review noted that the colours of the indicators have not always reflected the project health.

For example, TBS guidance notes that the overall Project Health indicator should change to “red” when there is, “one red indicator and at least one yellow indicator.” In November 2013, the Schedule indicator turned “yellow”, due to the re-baseline of the project schedule. Since April 2013, the Risk indicator had been reporting red. It turned “yellow” in November even though there were the exact same number and level of risks as reported in October 2013. As per TBS guidance, a red Risk indicator combined with a yellow Schedule indicator would have required the Project Health indicator to turn “red”. This was not the message the project wished to communicate. However, this was not a fulsome representation of project’s health.

In addition, the December 2013 Dashboard reported “yellow” Risk and Schedule indicators externally while internally the Risk and Schedule indicators were reported as red and the Issue indicator was yellow. Interviews indicated that project management required time to report on significant delays in the schedule to executive management, Treasury Board and external stakeholders. Although management is able to apply their discretion to the Dashboard indicators, if the TBS and CBSA-approved guidance on project health indicators is not followed or used correctly, Senior Management risks setting a precedent that sends the message to managers that they don’t need to highlight an issue that can be resolved without senior management’s intervention.

9.4.2 The full project schedule has not been reflected in project status reports

Based on the results of an in-depth review of eManifest conducted in September 2013 on the Associate Vice-President of ISTB’s, request and due to the short timeframe from when the Dashboard first reported overall project Health as “yellow” (May 2013) to when the schedule indicator turned “yellow” (November 2013), the audit team performed a comparison of various operational level reporting documents (e.g., IT schedule and Master schedule) to the higher level status reports (e.g., schedule status reports and Dashboard).

The comparison identified that concerns and slippages identified in the schedule status reports were, at a high level, reflected in the risks, issues or schedule sections of the Dashboard. It also confirmed that the information contained in the IT project schedule generally matched that in the Master schedule. However, a comparison of the Master Schedule against the Dashboard demonstrated some noteworthy inconsistencies. Up until January 2013, the Master Schedule identified that the project would be finished in April 2015, a delay of 4 months from targeted project completion, which was not considered by the project to be significant enough to change the Schedule indicator from “green”. However, in February 2013, the Master Schedule included a new section, labelled “final implementation,” which included changes to regulations, associated training and client support required after the system applications have been developed, and identified July 2016 as the date of completion for this section of the project. This was not reflected in the Dashboard.

When asked about the dates in the Master Schedule, project management noted that the dates reflected deliverables that will occur after the initial development and release of the system applications, such as the implementation of regulations, the communication and education associated to the regulations and support for both internal and external clients. Although this may not be required in reports to TBS, the audit team expected that the project status reports used internally, such as the Dashboard, would be reflective of the full scope of the project, not just the IT or development aspects. eManifest’s Project Charter identifies both “Implementation” and “Post-Implementation” as part of the project life-cycle, suggesting that the project will not be complete until both those phases are finalized. In addition, the Charter observes that, “the requirement for mandatory electronic reporting is dependent on regulatory and legislative changes,”Footnote 8 signifying that the project will not be fully implemented until the regulations have been rolled-out and enforced.

9.4.3 The Dashboard is not filled out correctly

As a part of the above-mentioned review, it was found that despite the information being consistently monitored and updated, information was regularly reported incorrectly on the dashboards. For example:

  • The risks identified in the “top risks” section were not always reflected in the risk matrix and the risk matrices didn’t always match the number of risks currently open on the risk log;
  • The schedule reported on the Dashboard often had dates for deliverables within a release that fall after the forecasted completion date of the release;
  • The “Estimated # of Effort Days,” “Estimated Cost” and “Impact on Project” sections for change requests have not been tracked as required by TBS guidelines; and
  • Change requests have been tracked differently over the course of the project – originally, all change requests were tracked, then no change requests were reported and now only requests that have affected the scope of the project are being tracked.

Filling out the dashboard correctly ensures that appropriate information is reaching the decision-makers and management of the Agency.

Recommendation 3:

The Vice-President, Information, Science and Technology Branch, in consultation with the Vice-Presidents Comptrollership, Programs, Operations and Human Resources, should ensure that all aspects of the scope of the eManifest project are reflected in the key project status reports and monitored until project completion.

Management Response

Management Response Completion date
Management agrees with this recommendation. The key missing piece that led to this recommendation was the absence of an end-to-end fully integrated plan. Each area had their own plans but there was not full integration by all areas responsible for project delivery. By transferring responsibility for full project delivery, accountabilities and controls to the Commercial Project Portfolio Director General, who is now responsible for producing and managing integrated plans, all controls are in place for effective and timely corrective action. This integrated project plan will be reviewed and monitored regularly and variances reflected in project status reports. The implementation of the CA Clarity tool will be leveraged to ensure there is alignment with the dashboard report and the integrated project plan. Project’s progress will be made visible within CA Clarity to the appropriate areas wherein they will be able to review the project’s progress via embedded status reports within the tool. July 2014

Recommendation 4:

The Vice-President, Information, Science and Technology Branch, should ensure that the eManifest Project Executive Dashboard is completed as per Treasury Board and Agency guidance, and is monitored for discrepancies until project completion.

Management Response

Management Response Completion date
Management agrees with this recommendation. The dashboard will be completed as per the Treasury Board guidelines. July 2014 (and ongoing)

Appendix A – About the audit

Audit objective

The objective of the Audit of eManifest Project Governance and Reporting was to determine whether the management processes, practices and controls in place are adequate to ensure that project governance and reporting is sufficient, accurate, complete, consistent and timely.

Audit scope

The scope of the audit covered project governance and all key project reporting. The audit included interviews and examination of project documentation that was developed during the period from August 1, 2012 to the end of the examination phase of the audit (February 2014).

The audit did not review the systems that will be developed as a part of the project or project management capabilities and practices, nor did it assess stakeholder management and benefits realization.

Risk assessment

This audit focused on areas of higher risk within project governance and reporting related to the eManifest. Areas that pose significant or high risk include:

  • There is a risk that the governance structures for the eManifest project are not appropriately established and functioning for timely and effective decision making and risk management.
  • There is a risk that the governance bodies do not receive accurate, complete, consistent, and timely project reporting for effective oversight.
  • There is a risk that project status reports do not accurately reflect project health regarding schedule, scope, deliverables, costs, issues and risks.
  • There is a risk that tolerance for risks and issues is not well defined, communicated, understood, and applied in project status reporting.

Approach and methodology

The audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada. The examination phase of this audit employed the following approach:

  • Reviewed the CBSA project management standards, guidelines, and directives related to project management reporting;
  • Conducted interviews with 14 stakeholders within the Programs, Operations, Comptrollership, and Information, Science and Technology branches on their roles and responsibilities, oversight function, and monitoring in relation to eManifest project governance and reporting;
  • Conducted six walkthroughs of the status reporting process with key stakeholders in Project Steering and Control Office;
  • Conducted a review of a sample of five (5) project dashboards to determine how information is incorporated;
  • Analyzed key sources of data for the dashboards;
  • Reviewed key documentation, such as the governance framework, terms of reference, records of decision, project charter, project schedule, risk, issue, and change logs logs, project dashboards, schedule status report, etc.

Audit criteria

Given the preliminary findings from the planning phase, the following criteria were chosen.

Lines of Enquiry Audit Criteria
Governance and Reporting Frameworks 1
  • 1.1  Effective oversight bodies are established and defined.
  • 1.2  Management has defined project status reporting requirements that address project schedule, scope, costs, deliverables, issues and risks, and reporting frequency.
  • 1.3 Risk and issue tolerance threshholds are defined, approved and communicated.
Decision Making 2
  • 2.1 Oversight bodies request and receive accurate, complete, consistent, and timely information to monitor eManifest’s progress.
Variance Monitoring 3
  • 3.1 Variances between actual and plan are analyzed and communicated, for project schedule, scope and costs.
  • 3.2 Management monitors the actual performance of eManifest against the planned results and adjusts course as needed.
Status Reports 4
  • 4.1 Project status reports are subject to management quality review and challenge for completeness, accuracy and consistency.

Appendix B – Reporting Tool Descriptions

eManifest Executive Project Dashboard (the Dashboard): The executive project dashboard is used to communicate key project information clearly and succinctly to departmental senior executives and others who play a governance or oversight role in a project. It includes information on the following:

  • Executive Summary
  • Financial Summary
  • Project Schedule
  • Project Scope
  • Project Risk
  • Project Issues
  • Overall Project Health

Attached to the Dashboard is a project summary document that provides more details on the different areas of the Dashboard.

Schedule Status Reports: Schedule Status Reports are updated every two weeks and provide information about status update/accomplishments, areas of concern that requires Senior Management assistance, action items to be addressed by the next update period, team schedule reporting and completion dates.

Master Schedule: The Master Schedule is an “evergreen” password-protected document that is updated when there is a change in the schedule. If the Master Schedule is updated, the remaining schedules must be updated at the same time.

Implementation Snapshots: Implementation statistics and trends are tracked throughout the year and the implementation snapshot was developed to offer management a picture of the activities occurring across the project. It includes information on the targeted outreach strategy, training and outreach activities (webinars and activities), transactions performed by officers in the field and engagement activities with carriers.

eManifest Project Benefits Report: The eManfiest Project Benefits Report presents a summary of the benefits that will be realized regarding the following three aspects: savings, security and service. The report provides details on the following aspects for those three main areas:

  • Financial Savings (including Cost Avoidance)
  • KPIs
  • Benefits Recipients
  • Risks (resulting in benefits not being fully realized)

Risk and Issue Logs: Risk and Issue logs are “evergreen” logs that track open and closed risks and issues that have arisen over the course of the project. Logs are updated based on information received from the governance committee(s) and/or updated by submissions from the project team, as risks or issues are identified. The logs are maintained and monitored by the PSCO.

Change Logs: Change logs are “evergreen” logs that track open and closed change requests that have arisen over the course of the project. Logs are updated based on information received from the governance committee(s) and/or updated by submissions from the project team, as required changes are identified. The logs are maintained and monitored by the PSCO.

Ad Hoc Reports: Ad hoc reports regarding eManifest are provided to management and governance committees, when requested, on various topics relating to the project.

Briefing Notes: Briefing notes regarding eManifest are provided to management and governance committees, when requested, on various topics relating to the project.

Appendix C – Governance Committee Descriptions

Business Managers Committee: This committee serves as the starting point for identifying and escalating items for decision making that pertain to any issue, risk, change request or other matters that pertain to eManifest. This committee is utilized to make recommendations to the appropriate Directors Committee.

Business and IT Managers Committee: This committee provides a forum for business and IT managers to discuss a high-level view of the current release status, including release content, problem reports and other challenges being encountered by both teams.

Project Directors Steering Committee: This committee acts as a forum to continuously monitor the progress of eManifest, including approving project dashboards. This committee guides the discussion and resolution for key items relating to the project, including issues, risks, change requests, schedules and any other matter that directly relates to both business and technical components.

Directors Program Alignment Committee: The overall mandate of the Directors Program Alignment Committee is to discuss and ensure progress against policy/program alignment issues and to ensure information is given to senior management for timely and informed decisions.

Directors General Program Alignment Committee: The overall mandate of the Directors General Program Alignment Committee is to monitor, guide and provide direction on program/policy alignment issues.

Directors General Projects Steering Committee: The overall mandate of the Directors General Projects Steering Committee is to monitor, guide and provide direction on project and policy issues.

Vice Presidents’ Major Projects Steering Committee: This committee plays oversight role and provides advice on the overall direction and progress of eManifest, including the accountability of adhering to project schedules and deadlines. This committee is governed by the Enterprise Project Management Office.

Senior Project Advisory Committee: This committee provides active advice on the overall direction and progress of the eManifest project and ensures alignment of the eManifest project with Government of Canada priorities and policies.

Appendix D – TBS Guidance on Project Health IndicatorsFootnote 9

Executive Summary:

This section does not have a status indicator.

Project Cost:

The Project Cost status indicator is automatically populated based on the variance between the Total Approved Project Budget and the Total Actual/Forecast Cost. This status indicator is locked and cannot be altered.

The Project Cost status indicator reflects the impacts to the budget being experienced by the project. Known changes to the project budget should be reflected in the forecast budget.

Any risks that may exist for potential changes to the budget should be reflected in the Risk section. Once a risk materializes, it is to be reflected in the Issue section. If it changes the actual or forecast cost, it will be reflected in the Project Cost status indicator.

Green indicates a variance of less than 10 percent. The project is expected to be completed within budget.

Yellow indicates a variance of between 10 and 20 percent. Project completion may not be possible within budget. Additional funding or re-profiling may be required.

Red indicates a variance of over 20 percent. Project completion may not be possible within budget. Funding decisions are required.

Financial Summary:

This section does not have a status indicator.

Project Schedule:

The Project Schedule status indicator is automatically populated, based on the variance between the Approved Project Completion Date and the Forecasted Project Completion Date, taking into consideration the Project Launch Date. This status indicator is locked down and cannot be altered.

Note: The Project Schedule status indicator reflects the impacts to the schedule being experienced by the project. Known changes to the project schedule should be reflected in the forecasted schedule.

Any risks that may exist for potential changes to the schedule should be reflected in the Risk section. Once a risk materializes, it is to be reflected in the Issue section. If it changes the forecasted schedule, it will be reflected in the Project Schedule status indicator.

Green indicates variance of less than 10 percent. Project completion is expected within the planned schedule.

Yellow indicates variance of between 10 and 20 percent. Project completion may not be possible within the planned schedule.

Red indicates variance of over 20 percent. Project completion may not be possible within the planned schedule. Scheduling decisions are required.

Project Scope:

The Project Scope / Requests for Change status indicator should reflect the impacts to scope being experienced by the project. Any risks that may exist for potential changes to scope should be reflected in the Risk section, not in this section. Once a risk materializes, it can be reflected in the Project Scope / Requests for Change status indicator.

Green indicates that the project is on track. Identified changes are not expected to negatively impact the project’s scope, cost or schedule.

Yellow indicates that some course correction may be required. Identified changes may negatively impact the project’s scope, cost or schedule.

Red indicates that significant course correction may be required. Identified changes may have a significant negative impact on the project’s scope, cost or schedule.

Project Risks:

Once risks are summarized in the risk matrix and the top three individual risks have been described, they are to be rolled up into an overall risk rating indicator. When interpreting the overall project risk indicator status, the project residual risks should be considered instead of its inherent risks. If residual risks have not been qualified or quantified, this indicator should represent the overall status of inherent risks.

If there is a significant difference between the picture presented by the numbers of inherent risks in the Inherent Risk Matrix and the overall indicator based on residual risk (e.g., there are very effectivemitigation strategies in place so the overall indicator is more positive than the matrix), this should be explained in the Appendix C: Project Summary Template that accompanies the dashboard.

The indicator, located in the upper-right corner of the Project Risk section, is to be interpreted as follows:

Green indicates that the project is on track. The risks are not expected to impact the other project metrics or overall business outcomes.

Yellow indicates that some course correction may be required. One or more identified risks may impact the other project metrics or overall business outcomes.

Red indicates that significant course correction may be required. One or more identified risks may impact the other project metrics or overall business outcomes.

Project Issues:

The Project Issues status indicator identifies the severity of the issues being experienced by the project and the degree to which they are being effectively resolved or acted upon. Any risks that mayexist should be reflected in the Risk section, not in this section. Once a risk materializes, it is to bereflected in the Project Issues section.

Green indicates that the project is on track. All identified issues are manageable within the project’s scope, cost and schedule.

Yellow indicates that some course correction may be required. One or more of the identified issues are potentially unmanageable within the project’s scope, cost or schedule.

Red indicates that significant course correction may be required. One or more of the identified issues are unmanageable within the project’s scope, cost or schedule, or the activities undertaken to date to resolve the issue(s) have not been effective.

Project Health:

Aggregate Indicators Overall Project Health
At least two red indicators Red
One red indicator and at least one yellow indicator
At least three yellow indicators
One red indicator and four green indicators Yellow
Two yellow indicators and three green indicators
Most other permutations Green

Applying judgment, adjust the current overall Project Health indicator up or down to reflect the realities of the project. This assessment of the overall project health requires the judgment of a senior level project manager or executive within the project governance body.

Green indicates that the project is on track.

Yellow indicates that some course correction may be required.hat significant course correction may be required.

Appendix E – List of Acronyms

ACI
Advanced Commercial Information
BMC
Business Managers Committee
BM&ITC
Business and IT Managers Committee
CBSA
Canada Border Services Agency
CMC
Corporate Management Committee
CPDSC
Commercial Project Directors Steering Committee
DGPSC
DG Project Steering Committee
EC
Executive Committee
EVM
Earned Value Management
ISTB
Information, Science and Technology Branch
IT
Information Technology
MPD
Major Projects Directorate
PMF
Project Management Framework
PoMP
Policy on the Management of Projects
PSCO
Project Steering and Control Office
RODD
Record of Discussion and Decision
SPP
Security and Prosperity Partnership of North America
TBS
Treasury Board of Canada Secretariat
TOR
Terms of Reference
VPMPSC
Vice-President Major Projects Steering Committee

Footnotes

Footnote 1

Adapted from the Policy on the Management of Projects, http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=18229&section=text

Return to footnote 1 referrer

Footnote 2

Journal for the Association of Information Systems, http://aisel.aisnet.org/jais/vol9/iss7/17/

Return to footnote 2 referrer

Footnote 3

A description of each tool is included in Appendix B.

Return to footnote 3 referrer

Footnote 4

Although TORs, RODDs and Agendas were reviewed for all committees listed in the governance structure in Figure 2, this section focuses on the committees in the Project Stream, as these manage and monitor project management information, such as cost, schedule and scope. No significant deficiencies were noted in the Policy Stream committee documents.

Return to footnote 4 referrer

Footnote 5

It should be noted that TORs and RODDs/Agendas were not reviewed for CMC and EC, as these committees’ mandates are broader than the management of projects or eManifest and were there considered to be out of scope.

Return to footnote 5 referrer

Footnote 6

This document describes how to develop an activity-based schedule, estimate activity duration, and apply earned value management to project deliverables.

Return to footnote 6 referrer

Footnote 7

From Deloitte review – Canada Border Services Agency Financial Review of Major Projects

Return to footnote 7 referrer

Footnote 8

Page 10, E-Manifest Project Charter, Issued September 6, 2006.

Return to footnote 8 referrer

Footnote 9

Taken from TBS’ Chief Information Officer Branch’s “Guide to Executive Project Dashboards for IT-Enabled Business Projects”, Version 3.0, issued July 2012.

Return to footnote 9 referrer

Date modified: