Tips for Successful AI Project Delivery

In a previous article, we learnt about how Artificial Intelligence/Machine Learning (AI/ML) projects are managed at AI Singapore (AISG). In this article, we will go through some of the lessons learnt from managing AI/ML projects in general.

What is expected of a Project Manager?

Typical expectations are: know your stakeholders and manage their expectations, continuous communication, manage the scope, get on top of the timeline, control the project cost, foresee all the pitfalls, be prepared to face off, etc. In my view, what is also important is what the project manager brings to the table in terms of the lessons learnt from past projects. This is where seasoned project managers stand out from the rest, but even so, they are not immune to the potential failure in foreseeing certain pitfalls, particularly when they are in a less familiar territory like managing AI/ML projects.

Managing AI/ML projects is not an art but science. Due to the dynamic nature of AI/ML projects, all the pitfalls a project manager faces while managing traditional software engineering projects, such as scope creeps, missed timelines, oversight in risk management and misaligned objectives, are amplified more here.

All right, how different are machine learning projects from software engineering projects? In traditional programming, with a set of rules/business logics built, the software applies the logic on the input data and produces an output. On the other hand, in machine learning, the ML algorithm automatically formulates the rules from the data.

From a project management perspective, software engineering projects are far better understood and mostly well documented whereas AI/ML projects are still evolving and in experimentation. The differences could derail an AI/ML project to a large extent if it were managed the same way as a software engineering project. Some of these differences are subtle enough that a mistake might not be apparent to the project manager until it occurs. Let us look at various potential pitfalls of managing AI/ML projects, in the area of stakeholders’ expectations, data and project timeline.

STAKEHOLDERS’ EXPECTATIONS – At the starting point of an AI/ML project, it is very important that all the stakeholders are aligned and are on the same page in terms of understanding the project objectives and the expected outcomes from the project. Some of the important questions to ask at this stage are:

1. Do the stakeholders understand the very nature of Machine Learning (or Deep Learning) processes?
2. Do the stakeholders have the same understanding of the definitions of ML model performance metrics like Accuracy, Precision, Recall, F1 Score, Mean Average Precision, etc., or do they know which metrics serve them and their businesses better?
3. Are the project acceptance criteria or project success metrics clearly defined with no ambiguity, i.e. what is expected from the AI/ML project?

For successful completion and implementation of any AI/ML project, the answers for all the above questions should be – Yes. We need to have a good understanding of the business case and ask the right questions to the stakeholders to:

1. Choose the appropriate ‘target’ dataset (list of features).
2. Choose the suitable performance metrics (Accuracy, Precision, etc.).
3. Be aware of and decide on the trade-offs between available choices.

For project acceptance, the project success metrics can be business-centric, e.g. improved productivity of X hours, but one should also express the expected performance of the ML model and the performance metrics to adopt, e.g. Y% Recall. To better manage the stakeholders’ expectations, it is suggested that the model performance be expressed as a range, e.g. (Y1 to Y2)% Recall. It is also suggested to qualify that, although a suitable model architecture plays a big part on the model performance, the model performance ultimately depends on the data, be it data quantity or quality.

One may want to consider the following additional non-functional acceptance criteria, along with the performance criteria:

• Model training and inference times.
• How frequent the ML model is to be trained or utilized – this would help indicate how crucial are the model training and inference times.

DATA, DATA AND DATA – Data is key, in terms of its availability, quantity and quality for the training and evaluation of the ML model, and it is central to the success of AI/ML projects. All ML algorithms depend on a sizable set of data to train the model and the accuracy of the model largely depends on the quality of the training data.

Data Readiness – It is imperative that the required size and quality of data is available at the onset of AI/ML projects. This enables critical data related activities to be performed, such as data checking, exploratory data analysis, etc., according to the project timeline. It is from such activities that data quantity and/or data quality issues, if any, are discovered. Early discovery of data issues allows remedy actions to be followed up to continue to steer the project forward. To better pre-empt data issues, it is recommended to obtain some samples of the data, prior to the commencement of the project, to perform sanity checks.

Data Volume – A large dataset, in the magnitude of thousands of data points, if not more, is needed in most cases of AI/ML projects. Shortfall of the required data volume is likely to result in poor or decreased performance in the ML model accuracy and consequently unsatisfactory customers. To enable the model training, especially of deep learning models, which involve large datasets, it would require reasonably high-end compute and storage resources, so one has to be prepared to have the required GPUs and gigabytes of storage space up and ready.

In the case where the large dataset is extracted from multiple data sources and/or of various data formats, one also has to be prepared for the potential increase in the data extraction, transformation and ingestion effort, as extracting data from different sources will require the data pipeline to set up many more connections and different interfaces (e.g. web crawling, APIs, etc.) which is time and effort consuming. This has cost and time implications to the project.

Data Quality – Data quality issue encompasses many different aspects, such as invalid data format (e.g. wrong datetime format), incomplete data (e.g. missing values), noisy data (e.g. outliers), imbalanced data (i.e. the distribution of examples across the known classes is biased or skewed), unlabeled data or missing annotations, unscaled data, etc. Poor data quality, if not treated, will impact the ML model performance, regardless of the model architecture employed. Having said that, one has to be aware of the additional time and effort needed to treat data (data analysis and visualization, data cleaning, data imputation, etc.) and cater for them in the project timeline.

PROJECT TIMELINE – Failure to plan is planning to fail. It is understandable that stakeholders expect a short implementation timeline for projects, but it is important to note that AI/ML projects may require a special degree of patience and flexibility. First and foremost, during the initial stage of a project, if no minimum training data (adequately annotated) is available and it cannot be made available very soon, the project is set to be delayed. Without the training data, work related to exploratory data analysis, feature engineering and ML model training will be impeded. In such a situation, it is better off to postpone the project.

Even if the training data is sufficient and ready, due to the nature of AI/ML projects where a lot of experimentation must be carried out before a suitable algorithm is zeroed down, enough time must be considered in the project timeline for the various experiments. Also, the project progress could well appear to be slow initially, so one must constantly educate and update the stakeholders on the incremental work completed, to assure them that the project is progressing to meet the project objectives according to the timeline.

It is also important to note that, during the acceptance stage of a project, if the stakeholders are not aligned with the project outcomes and, while testing the ML model, they start to develop unnecessary assumptions and expectations on what the model can deliver or cannot deliver, the testing and acceptance process will be dragged and the timeline will be impacted. This point ties back to stakeholders’ expectations and can be mitigated if the stakeholders’ expectations are properly managed.

IN CONCLUSION, this article aims to share and highlight the pitfalls of managing AI/ML projects in the area of stakeholders’ expectations, data and project timeline. Knowing the pitfalls and being conscious of them helps a project manager a long way in managing and delivering AI/ML projects successfully.

The AI Project Management Series

• Managing AI Project in AISG
• Lessons From Managing AI Projects (this article)