Canada Border Services Agency
Symbol of the Government of Canada

ARCHIVED - eManifest Review

Review Engagement

Warning This page has been archived.

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

December, 2012

This document is also available in PDF (653 KB) [help with PDF files]


Table of Contents



Return to Top of Page

1.0 INTRODUCTION

eManifest is a major crown project underway at the Canada Border Services Agency (CBSA). Once fully implemented, eManifest will change the commercial import process. It will require members of the trade community such as carriers, freight forwarders, and importers in all transportation modes to electronically transmit data on cargo, conveyance and crew to the CBSA before shipments arrive at the border.

The Internal Audit Division completed a review of the eManifest project which included controls over the project scope, project management practices, cost management processes and project status reporting. The review included interviews and examination of project documentation that took place during the period from January 1, 2012 to August 30, 2012.

2.0 SIGNIFICANCE OF REVIEW

This review is significant because large government Information Technology (IT) projects like eManifest are inherently complex, expensive and risky, and have had a history of cost overruns and delays, and not delivering what had been planned. This review of eManifest will provide CBSA management with information on the adequacy of its project management practices, in general and for the eManifest project.

OBJECTIVE

The objective of this review was to determine the extent to which eManifest will meet the needs of end users; the adequacy of the project plan with respect to managing the project’s scope, schedule and resources; the management of costs; and the effectiveness of tools and processes for status reporting.

3.0 STATEMENT OF CONFORMANCE

This review conforms to the Internal Auditing Standards for the Government of Canada, as supported by the results of the quality assurance and improvement program. The approach and methodology for this review followed the International Standards for the Professional Practice of Internal Auditing as defined by the Institute of Internal Auditors and the Internal Auditing Standards for the Government of Canada as required by the Treasury Board Internal Audit Policy.

This review was intended to provide a low level of assurance commensurate with a review.

Return to Top of Page

4.0 KEY FINDINGS

Consultations are occurring with the future end users of eManifest. This is being done through project working groups and governance committees. Some end users have expressed an interest in having improved discussion on business requirements and feedback on decisions taken. Project management understands the importance of good stakeholder consultations and considers this a low risk in the overall context of the project.

The eManifest project has performed multiple types of testing on previous releases. However, at the CBSA, standards for test plans have not been developed and final acceptance testing does not include end users. Test results have not been shared with stakeholders to allow for a more informed review and evaluation of the system as implemented. Ineffective acceptance testing could lead to issues and change requests after the system is in the production environment.

The trade-off between project efficiency and end user involvement presents risks that need to be considered and weighed. For example, the initial release of Build 1 for March 31, 2013 could be at risk. The preliminary estimated effort to conduct testing was greater than the time available. If this becomes an issue, management will have to weigh the trade-offs before making a decision. Project governance committees at all levels are in place to facilitate the decision-making process.

For project planning and monitoring, management does not have a complete set of project management information to monitor progress with respect to time, scope and cost. Each individual manager is responsible for managing and reporting monthly on their piece of the project. There is reliance on this reporting process and confidence that issues and risks will be identified and addressed in a timely and effective fashion.

Earned value management (EVM) is a process that provides management with information on investment performance and necessary data points to estimate probable completion costs and dates. As part of the Agency’s cost management plan for eManifest, it had agreed to use EVM to measure performance. While this work was initiated, it was discontinued due to the complexity and lack of agency tools for the EVM process.

Multi-year projects are expected to forecast costs based on a costing template that aligns with the Treasury Board Submission. eManifest is not following this process. Project managers are negotiating their budget requirements with the eManifest project office without knowing the cost of the detailed project plans. Cost forecasts to complete eManifest are based on the original Treasury Board submission and workplans developed by project managers.

Comptrollership Branch has indicated that it is unable to provide reasonable assurance on the project’s financial health because project costs and forecasts are not done by deliverable. Comptrollership does monitor and discuss actual cost information with project management to ensure it is consistent with corporate information.

Project management is reporting regularly on the status of the project. Clarification is required on the information that should be reported, including the completeness of some of the dashboard elements. This includes reporting by deliverable, rather than project phase. Oversight on the dashboard could be strengthened to ensure all significant issues from CBSA stakeholders are reported. The risk is that CBSA executive management may not be informed of significant issues that could have broader implications for operations.

Return to Top of Page

5.0 SUMMARY OF RECOMMENDATIONS

The review makes six recommendations to:

  • Review the current consultation process with end users to ensure it is understood and issues are properly addressed.
  • Develop and implement CBSA project standards specific to test plans and user acceptance testing (UAT).
  • Assess whether the system development life cycle and project management practices that support agile techniques undertaken by the eManifest project are acceptable and applicable to other Agency projects and update project management policies and guidance accordingly.
  • Develop and implement a process to demonstrate benefits realization of eManifest.  
  • Provide assurance on the financial health and the remaining cost forecast of the eManifest project and sign off as such on the dashboard.
  • Require that the Director General (DG) Projects Steering Committee reviews and discusses all project risks and issues and include significant risks and issues on the eManifest project dashboard.

6.0 OVERALL MANAGEMENT RESPONSE

The Programs Branch, Comptrollership Branch, and the Information, Science and Technology Branch (ISTB) agree with this review and its findings.

Programs Branch will implement an improved consultation process with end users, will demonstrate the benefits realization of eManifest, and will take steps to ensure that the DG Projects Steering Committee is able to review and discuss all significant project risks and issues.

ISTB will assess whether the system development life cycle and project management practices that support agile techniques, undertaken by the eManifest project are acceptable and applicable to other Agency projects and will update project management policies and guidance accordingly. ISTB will also move forward with a modernization and transformational proposal for a testing framework/approach to address the recommendations.

Comptrollership, in consultation with Programs Branch (Office of Primary Interest), ISTB (Office of Collateral Interest) and the Enterprise Project Management Office (EPMO) will implement a process to strengthen the quality assurance of project financial information and the remaining cost forecast of the eManifest project and sign off as such on the dashboard.

Return to Top of Page

7.0 FINDINGS

7.1 Scope and user requirements

Project Management Objective:  The project will meet its scope and end user requirements.

The success of a project depends on the effectiveness of the final project output in meeting its initial business objectives and requirements of the business process owners and end users. Good project management practices assist project managers to meet the project scope and end user requirements. These practices include a sign-off of the scope definition and user requirements, having appropriate safeguards in place to manage changes to project scope definition and user requirements, and conducting user acceptance testing before the system is released into the production environment.

Project Scope:

Prior to 2010, sponsor branches (Admissibility, Enforcement and Operations branches) approved business requirements. As of the April 1, 2010 reorganization, the Major Projects Directorate of Programs Branch assumed the role of Office of Primary Interest (OPI) for the eManifest project. Offices of Collateral Interest (OCI), including other directorates within the Programs Branch, and other branches, now serve the role of Subject Matter Experts (SMEs). The purpose of this reorganization was to improve the governance structure efficiencies and to streamline the escalation process. As a result, internal end users (Operations Branch and Programs Branch, other than Major Projects Directorate) are no longer required to sign off business requirements and they no longer have the approval authority on policy issues and scope definition. The trade-off between efficiency and end-user involvement presents itself with different risks that need to be considered and weighed.

Currently, the Commercial Unit of the Pre Border Programs Directorate is tasked with coordinating consultations on requirements between the project team and representatives of future end users. The Commercial Unit identified relevant OPIs and provides relevant documentation for comments. Once comments are received, the Commercial Unit will summarize them and attempt to find a solution to issues. If an issue cannot be resolved, it is sent for a decision from the Director Policy Steering Committee. Other avenues for feedback include an internal e-mail box, where users can send issues for the policy team to review and action accordingly. The process has since evolved and improved through the creation of the Managers Policy Committee in February 2012. This committee was created to review new system functionality and discuss policy issues every two weeks.  These sessions should provide an opportunity to discuss and validate user needs. Although consultations are occurring with future internal users, some end users expressed an interest in having improved discussion on business requirements and feedback on decisions taken. Project management understands the importance of good stakeholder consultations and considers this a low risk in the overall context of the project.

Project Change Controls:

Regarding safeguards to manage changes to the project scope definition and end user requirements, the eManifest project has adopted a more iterative approach to managing the project. The approach used by the eManifest project team is inspired by agile project management concepts and is a recognized approach to project management in the industry. Agile requirements elicitation requires regular consultations with users through various methodologies to accommodate acceptable business demands.

The eManifest project teams and information technology representatives must agree on which project artefacts define the scope and the business requirements associated with each of the remaining three builds that will form the basis for the development of systems. In most cases, the business teams within the eManifest project are producing user requirements specifically aligned to each build. An agreement on requirements forms the basis of the baseline, for which systems are developed. The baseline for the first two builds was established in July 2012 creating a risk that IT teams had been working using project artefacts that were not finalized. A requirements baseline is an important element of change control, as all subsequent modifications to requirements after the baseline is established require a formal change request. Without a baseline, user requirements may fluctuate during the development process and cause unwanted consequences such as rework to developed systems.

An additional consideration is the requirement for collaboration with future users during requirements elicitation. As noted, the future three builds have incorporated the more iterative agile approach to requirements definition. The current approach adopts aspects of an agile approach with some important deviations; the eManifest project does not employ smaller release cycles, in which the product is continuously ready for release. Additionally, continuous feedback between end users and the development teams is not carried out. Consequently, there can be important lags between when comments are received from stakeholders and when these groups will validate the functionality developed.

Return to Top of Page

User Acceptance Testing:

Typically, before system changes are released into the production environment, the application is tested in a controlled environment to determine functionality and usability using scenarios that resemble how the application will be employed by the end user.

The eManifest project has a testing plan and testing of previous releases has occurred. However, at the CBSA, organizational guidance or standards specific to test plans or user acceptance have not been developed.

With respect to testing activities for eManifest releases, areas to strengthen would include more involvement of end users, and improved documentation and communication of test results. Specifically, end users were not involved in acceptance testing and there was no separation of duties between the individuals who defined the requirements and those who tested the requirement. The test plan was insufficient in detail. Test plans should include details on the scope of testing, schedule, test deliverables, release criteria and risks, and contingencies. The test results reviewed were not adequately documented and not shared with all stakeholders to allow for an informed review and evaluation of the system implemented by internal project stakeholders such as Operation Branch, Border Programs and Pre-Border Programs.

In May 2012, a release pertaining to the Passage Deployment for Rail had a reduced testing period. There were problems with the implementation of the system and resolution of these issues created a delay of one month to the development schedule for the next release. The May 2012 release proceeded to meet project time commitments. It is recognized that management needs to balance project scope requirements, cost and schedule and manage the risks associated with them.

Summary

In regard to this project management objective as noted above, the Agency is exposed to risks that system functionality may not meet the initial needs of end users and additional costs may be incurred to make change requests to respond to users’ concerns. As well, expected results to enhance employees’ efficiency may not be achieved and management may not be able to obtain the information it requires to effectively manage the business process.

Within the project, there are different levels of risk tolerance. The trade-off between project efficiency and end-user involvement presents risks that need to be considered and weighed by the project governance committees.

Return to Top of Page

RECOMMENDATION 1

The Vice-President of Programs Branch should review the current consultation process with end users to ensure the process is understood, and end user feedback is considered and issues are properly disposed of.

Management response

Programs Branch agrees with this recommendation. The eManifest project team, in consultation with Operations Branch, will enhance stakeholder consultations by conducting focus groups with end users on end state passage and risk assessment systems. It will also review and document the current process for consultation with end users, and will identify any significant gaps in this process. By March 2013, Programs Branch will implement a revised consultation process that will ensure the process is understood, the views of end users are considered, and any issues identified by the process are addressed appropriately.

RECOMMENDATION 2

The Vice-President of ISTB in collaboration with the Vice-President of the Programs Branch should develop and implement CBSA project standards specific to test plans and user acceptance testing (UAT).

Management Response

The Information, Science and Technology Branch (ISTB) agrees with this recommendation. ISTB will move forward with a modernization and transformational proposal for a testing framework/approach to address standards specific to test plans and UAT. By April 2013, a Testing Model Pilot will be implemented.

Return to Top of Page

7.2 Project Plan

Project Management Objective:  The project plan is sufficiently detailed to guide the project throughout its lifecycle.

Project management guidance at the CBSA has established a common approach for managing projects. Project plans are required documentation that includes scope, deliverables, work breakdown structures, roles and responsibilities, cost estimates, milestones, key dependencies and the identification of the project's critical path. An effective project plan establishes a baseline for the project and enables management to track progress in terms of scope, time and costs. This guidance developed by the Enterprise Project Management Office is largely based on traditional project management principles.

The eManifest approach to project scheduling adopts some agile concepts to realize efficiencies. The Agency’s guidance does not provide for alternative methodologies such as agile project management, which is recognized as a valid project management methodology. In summary, the eManifest project management approach includes:

  • A scope definition that aims to realize efficiencies. In December 2010, the eManifest Strategic Deployment View Task Force was established to develop a high-level deployment sequence for the remaining eManifest deliverables. This scope definition was amended to allow deliverables to be released in parallel and realize efficiencies in the schedule.
  • A time box approach was adopted for project planning, where deadlines (end dates) related to system implementation are fixed. The scheduled dates for tasks are permitted to fluctuate to meet deadlines. Each project team is accountable for meeting its deadlines.
  • The master schedule contains the duration and estimated hours per task; a high number of detailed plans are aggregated to the project master schedule. The granular project plans contain the deliverables agreed upon in the scope definition document.
  • Three Builds for eManifest remain with release dates; for Build 1 – March 2013, Build 2 – July 2014 and Build 3 – December 2014.

In project planning, a work breakdown structure helps accurately define and organize the scope of the project by breaking down the project deliverables into work packages across different levels of the project plans. Based on a review of project plans, the following was observed:

  • Project deliverables were indicated on detailed project plans, 
  • Activities have been assigned an estimated duration, but did not provide specifics on resource assignments, estimated effort or standard costs,
  • For detailed project plans, the 8/80 rule was not employed; this guideline stipulates that no work package should be less than 8 hours or greater than 80 hours. This rule allows project managers to govern the size and scope of the smallest chunks of work and apply consistent rules across work breakdown structures.

A critical path method is used to predict project duration by analyzing the sequence of activities that has the least amount of scheduling flexibility. For eManifest, the traditional critical path method was not employed. The eManifest project schedule assumes no flexibility in delivery for builds.

The initial release management timelines for Build 1 could be at risk. The preliminary estimated effort to conduct testing was greater than the time available to meet the delivery date of March 2013. Project governance committees will need to engage if potential issues arise in order to facilitate decision making.

Summary

With respect to the current project plan for eManifest, management does not have a complete set of project management information and as such it limits management's ability to monitor progress with respect to time, scope and costs. Each individual manager is responsible for managing his/her piece of the eManifest project including resources, scope and schedule. Managers are required to report regularly on the status of their component. Once a month, the results from managers are consolidated into the overall eManifest project dashboard. There is considerable reliance on these management processes and confidence that issues and risks will be identified and addressed in a timely and effective fashion.

RECOMMENDATION 3

The Vice-President of ISTB should assess whether the system development life cycle and project management practices that support agile techniques, undertaken by the eManifest project are acceptable and applicable to other Agency projects, and update project management policies and guidance accordingly.

Management Response

The ISTB agrees with this recommendation. ISTB will assess whether the system development life cycle and project management practices that support agile techniques, undertaken by the eManifest project are acceptable and applicable to other Agency projects, and update project management policies and guidance accordingly. The planned completion date is April 2013.

Return to Top of Page

7.3 Cost Management

Project Management Objective: Costs are monitored; deviations are identified and addressed in a timely manner.  Performance will be measured and validated.

Under Treasury Board Policy on the Management of Projects, projects are expected to achieve value for money and deliver outcomes within cost constraints. The Agency also expects that projects achieve value for money; the CBSA submitted a cost management plan as part of the effective project approval that stated that the earned value management technique would be used to ensure that costs were managed effectively, and performance was measured and validated.

Earned value management (EVM) is a process that provides project managers and stakeholders with insight into investment performance, and provides the necessary data points to estimate statistically probable completion costs and dates. The implementation of EVM is a recognized parameter of good project management. It ensures that cost, schedule, and technical aspects of the investment are integrated and:

  • Relates time-phased budgets to specific tasks;
  • Indicates work progress;
  • Properly relates cost, schedule, and technical accomplishment to scope;
  • Is valid, timely, and auditable;
  • Allows for statistical estimation of completion costs;
  • Supplies managers with information summarized at a practical level;
  • Facilitates risk management; and
  • Provides information on benefits realization.

For eManifest, EVM techniques are not currently being employed. Project managers indicated that it was difficult to meet EVM requirements with the tools they presently have.

Currently, the eManifest project team is tracking actual costs through CAS. Different groups however are reporting their costs differently. For example, ISTB employees use internal orders to report their time. Other groups do not employ internal orders for their timesheets; rather they perform a salary allocation to the project based on actual or estimated costs. The internal order code is not a mandatory field in CAS. As such this creates a risk that costs are not distributed accurately by projects.

At present, it is difficult to relate the full project cost to the actual and future project deliverables and builds. Multiple systems are employed to support the cost management process such as COMPASS and CAS but are not integrated with the eManifest project plans.

Multi-year projects are expected to forecast costs based on a costing template that aligns with the Treasury Board submission. eManifest is not following this process. Project managers are negotiating their budget requirements with the eManifest project office without knowing the cost of the detailed project plans. Cost forecasts to complete eManifest are based on the original Treasury Board submission and workplans developed by project managers.

As per the CBSA project guidance, the Comptrollership Branch and the EPMO have important roles related to cost management. The EPMO is responsible for providing project oversight, maintaining project management guidance, reviewing and analyzing the dashboard reports to ensure accurate reporting on key project metrics, and validating and challenging the dashboard information. Comptrollership is responsible for providing and completing the project financial information and for performing quality assurance of project financial information. Comptrollership is unable to provide assurance on project financial health because the project costs and forecasts are not done by deliverables. Comptrollership does review, monitor and discuss with project management the cost information to ensure the financial information aligns with CAS and COMPASS.

Summary

In regard to the project management objective, the Agency is exposed to the risk that CBSA management may not be made aware in a timely manner of potential cost overruns to the eManifest project. Actual costs are monitored and the eManifest burn rate is tracked to the original eManifest budget.

Management recognizes these challenges to integrate all costs and forecasts by deliverable and relate them back to the project benefits. EVM is one approach to accomplish this but requires the capacity, tools and discipline to implement.

Return to Top of Page

RECOMMENDATION 4

The Vice-President of Programs Branch should develop and implement a process to demonstrate the benefits realization of eManifest.  

Management Response

Programs Branch agrees with this recommendation. The eManifest project will complete a draft report demonstrating the benefits realization of the eManifest project by January 2013. The project will then submit this report to the Vice-President-level Projects Steering Committee for approval by March 2013.

RECOMMENDATION 5

The Vice-President of Comptrollership Branch should provide assurance on the financial health and the remaining cost forecast of the eManifest project and sign off as such on the dashboard. This includes a discussion with the Programs Branch on the financial information requirement to provide the necessary assurance.

Management Response

The Comptrollership Branch agrees with this recommendation.

Since the eManifest Review, Comptrollership has worked collaboratively with key stakeholders (OPI, OCI and EPMO) to establish the necessary elements to allow for the provision of quality assurance on the financial health.

An Internal Order structure (broken down by Components and Builds) has been established and agreed upon. Costs and timelines will be assigned to each Component and Build to further enhance the ability to monitor and report on the financial health, commencing April 1, 2013. This measure will help the Agency advance towards the principle of Earned Value Management (EVM).  

The Branch Chief Financial Officer will be signing off, and/or providing comments, where necessary, on the monthly dashboards to confirm that an assurance on the financial health, including the remaining cost forecast, has been conducted.

Return to Top of Page

7.4 Project Status Reporting

Project Management Objective: Tools and processes for status reporting are used appropriately.

eManifest project management is required to report the project status to CBSA senior executives and the Treasury Board Chief Information Officer Branch on a monthly basis. This is accomplished through a project dashboard designed to report on the core project metrics of cost, schedule, scope, risks and issues. Data requirements are stipulated for each metric within the Guide to Executive Project Dashboards for Treasury Board Oversight Projects. For example, to report on the project schedule, project management specifies the approved completion date, forecasted completion date and variance by deliverable. According to the Treasury Board Secretariat’s guide, the variance indicator is calculated based on the difference between the revised completion date (approved) and the forecasted completion date.

The status of each metric is colour-coded using green, yellow and red indicators. Green indicates a variance of less that 10%, yellow is a variance between 10 and 20% and red is a variance of over 20%. The eManifest Project Support and Control Office completes the project reporting process by collecting the information from various systems, and documents and validates the information with the Project Director's Steering Committee before it is inserted on the dashboard. As noted above, the quality assurance of financial information is to be provided by the Comptrollership Branch. The EPMO reviews the reports and provides a challenge function on the accuracy and completeness of the reporting. The final dashboard is signed off by the Major Project’s Director General.

The guidance on roles and responsibilities is sufficiently defined and key stakeholders understand their roles and responsibilities for approvals and quality reviews.  

For eManifest, there are some deviations from Treasury Board Secretariat’s guidance for reporting on the project schedule. The schedule section of the status reports asks that variances be reported for key milestones or deliverables based on the approved completion date and forecasted completion date. For eManifest, the EPMO has raised concerns regarding the definition of a milestone or deliverable. Currently, the project management uses project phases (e.g. construction and testing) as their definition for milestones or deliverables. The EPMO requests that the dashboard reflect key deliverables by the remaining Builds. This would be more effective in monitoring progress and potential slippage.

Additionally, the calculation of variance (slippage as per the Treasury Board Secretariat and EPMO guide) deviated from the policy. Project management currently uses a project schedule that includes the entire project development time for eManifest (2007 to 2014). The current project schedule was baselined and approved in January 2012 with final project implementation by December 2014. Therefore, the slippage calculation should be based on the revised baseline project duration.

During the course of the review, various stakeholders raised a couple of issues and they appear to be of significance for reporting on the project dashboard:

  • Key regulations that needed to come into force by November 2012 to enforce the mandatory requirement for highway carriers to transmit information electronically are not in place. As a result the anticipated benefits have been delayed.
  • Related to the first point, there is a slow uptake of highway carriers to employ electronic data interchange. As of October 2012, 3,339 out of 15,000 active carriers were registered which account for 22%, however it represents 77% of total trade volumes. Of the remaining non-registered carriers, 5,651 carriers had averaged less than one release per month of which 1,381 had only one release during the year. This could potentially create a demand from carriers when the mandatory requirement is put in place which the agency is unable to deal with in a timely matter.

Regarding project risks, project risks are identified on the dashboard and they include mitigation strategies. In reviewing some of the mitigation strategies, it is questionable whether the mitigation will achieve a desired outcome in response to the risk. For example, a project risk identified is the resource capacity of the ISTB to deliver on eManifest in addition to other Agency projects such as the Beyond the Border Action Plan and the CBSA Assessment and Revenue Management initiative. The mitigation was that eManifest is the foundation of many initiatives and the Agency’s commitment is to continue to implement eManifest and secure required resources. The review team has heard that this risk is now manifesting itself. As such this should be noted as an open issue and managed accordingly.

Summary

In regard to this project management objective, project management is reporting regularly on the status of the project and the tools and dashboards are being used. Clarification is required on the information that should be reported in the dashboard and the completeness of some of the dashboard elements, particularly the risk and issues section. Oversight on the dashboard reports could also be improved to ensure all significant information from CBSA stakeholders is reported. The risk is that CBSA executive management may not be informed of significant issues that could have broader implications for operations.

Recommendation 6

The Vice-President of Programs Branch should require the DG Projects Steering Committee to review and discuss all project risks and issues, and include significant risks and issues on the eManifest project dashboard.

Management Response

Programs Branch agrees with this recommendation. By January 2013, detailed risk and issue logs will be standing items on the DG Projects Steering Committee agenda where they will be reviewed and discussed, and significant items will be included on the Project Dashboard.



Return to Top of Page

APPENDIX A – ABOUT THE REVIEW

REVIEW OBJECTIVES AND SCOPE

The objective of this review was to determine the extent to which eManifest will meet the needs of end users; the adequacy of the project plan with respect to managing the project’s scope, schedule and resources; the management of costs; and the effectiveness of tools and processes for status reporting.

The review scope included all aspects of the eManifest project; there were no exclusions. The engagement was a review and not an audit as detailed testing was not undertaken. The level of our work was commensurate with a review, which provides a low level of assurance.

RISK ASSESSMENT

A number of risk areas were identified in the planning phase of the review:

  • Scope:  In the absence of adequate controls over scope management there are risks that the project will fail to meet business and user requirements leading to the rejection of delivered capabilities. Additionally, there is a risk that insufficient testing will result in unsatisfactory systems to business users.
  • Project Management: In the absence of reasonable project management practices, there are risks of undetected deviations in scope, schedule, and costs. Additionally, without consistent project management practices there are risks of inconsistency across interdependent projects.
  • Cost Management: In the absence of sufficient controls over costs, there are risks of a lack of control over project costs resulting in misspending of IT investments.
  • Project Reporting: In the absence of satisfactory controls over project reporting there are risks that reporting is ineffective on project progress which may lead to unidentified issues, undetected project showstoppers or undetected project risks. There is also a risk of loss of focus on customer expectations and business needs.

APPROACH AND METHODOLOGY

The review was based mainly on interviews and documentation reviews using two frameworks:

  • Control Objectives for Information and Related Technology (COBIT) version 4.1; and
  • Project Management Body of Knowledge (PMBOK).

COBIT identifies the IT processes that should exist to ensure that IT is aligned with and supports the business in an effective manner. COBIT and its supporting publications identify control objectives, techniques and practices commonly required for each process.

PMBOK identifies the best practice process for project management, together with the knowledge and techniques required for those processes to be effective. PMBOK can be designed to be applied to any industry, including IT.

Return to Top of Page

THE REVIEW CRITERIA USED FOR THE ENGAGEMENT PHASE WERE (by lines of enquiry):

1. Scope and user requirements
Criteria

The project will meet its scope and the requirements of end users.

Subcriteria

Project Scope: The project scope is defined and formally approved by the program and project sponsors.

Project Change Control: The project plan reflects changes to scope for approved changes, a baseline is established and the project plan is updated to reflect approved change requests.

User acceptance testing: The system meets the requirements of the business end users through successful user acceptance testing.

2. Project Plan
Criteria

The project plan is sufficiently detailed to guide the project throughout its life cycle.

Subcriteria

Integrated Project Plan: The scope, deliverables, required resources and responsibilities, work breakdown structures, estimates of resources required, milestones, key dependencies, and identification of a critical path are documented in the integrated project plan.

3. Cost management
Criteria

Costs are monitored; deviations are identified and addressed in a timely manner.

Subcriteria

Cost Management: A cost management process has been implemented comparing actual costs to budgets. Costs are monitored and reported, where deviations are identified and assessed in a timely manner.

4. Project Status Reporting
Criteria

Tools and processes for status reporting are used appropriately

Subcriteria

Roles and responsibilities related to status reporting are clear and understood.

Project Reporting: Project status reporting complies with the relevant organizational standards.



Return to Top of Page

APPENDIX B – GLOSSARY

Acceptance testing:
Applying performance and capability measures to project deliverables to ensure that they meet specifications and requirements and satisfy the customer.
Agile Project Management:
This is a highly iterative and incremental process, where developers and project stakeholders actively work together to understand the domain, identify what needs to be built, and prioritize functionality. Agile methods are used when these conditions are present: project value is clear; the customer actively participates throughout the project; the customer, designers, and developers are co-located; incremental feature-driven development is possible; and visual documentation (cards on the wall vs. formal documentation) is acceptable.
Baseline:
Original Plan (for a project, work package, activity), plus or minus any approved changes. May be used with a modifier (for example, cost baseline, schedule baseline, performance measurement baseline).
Baseline Measurement:
Set of approved estimates for scope, cost and schedule that forms the basis for change management during project execution. A project is baselined at the end of the planning phase.
Build 1:
Functionality that allows freight forwarders in all modes of transportation to transmit pre-arrival detailed data (i.e. House bill) is expected for March 2013.
Build 2:
Functionality for commercial passage and core risk assessment is expected for July 2014. It will also include the functionality to allow importers to transmit Advance Trade Data.
Build 3:
Enhancements to Risk Assessment and Passage are expected in December 2014, and will include enhancements to the commercial passage and risk assessment functionality, as well as new functionality for crew risk assessment in all modes.
Milestone:
1. An event with zero duration and requiring no resources. Used to measure the progress of a project or program and signify the completion or start of a major deliverable or other significant metric such as cost incurred, hours used or payment made.
2. Identifiable point in a project, program or set of activities that represents a reporting requirement or completion of a large or important set of activities.
Project Scope Management:
Includes the processes required to ensure that the project includes all the work required, and only the work required, to complete the project successfully. Project scope management is primarily concerned with defining and controlling what is and what is not included in the project.


Return to Top of Page

APPENDIX C – ACRONYMS

CARM — CBSA Assessment and Revenue Management

CAS — Corporate Administrative System

CBSA — Canada Border Services Agency

COBIT — Control Objectives for Information and Related Technology

COMPASS — Compliance Measurement, Profiling and Assessment System

DG — GDirector General

EPMO — Enterprise Project Management Office

EVM — Earned Value Management

ISTB — Information, Science and Technology Branch

IT — Information Technology

OCI — Office of Collateral Interest

OPI — Office of Primary Interest

PMBOK — Project Management Body of Knowledge

SME — Subject Matter Expert

UAT — User Acceptance Testing