AI experts talk to PMI Manage South Asia about the hurdles riddling DS/AI, and how to navigate them

The year 2020 was replete with instances of some of the biggest success stories and upsets for artificial intelligence (AI). The biggest hypes came from Elon Musk’s announcement of Neuralink, a brain hacking implant, GPT-3 by OpenAI that may soon eliminate our ability to distinguish a text produced by machines, and the widespread use of facial recognition around the world in the wake of COVID-19. However, there have also been some stunning failures – an algorithm giving incorrect predictive scores for school-leaving students in the United Kingdom and predictive models producing incorrect results, thus putting sales forecasts and inventory management during the pandemic into a tailspin.

It goes to show that while there has been tremendous progress, failures are common too. The high failure rate in Data Science (DS) and AI projects is in fact leading to frustrations and re-thinking on AI investments.

Last year, PMI collaborated with the Center of Excellence, DS & AI of the National Association of Software and Services Companies (NASSCOM), to publish a playbook on what lies behind the success or failure of DS/ AI projects. The playbook recommends a fit-for-purpose framework for DS/AI projects. It also puts together insights gleaned from surveys and interviews with DS/AI leaders from 25 organizations across industries, geographies and types of organizations.

As many as 88 percent of organizations covered in the study reported gaps in their practices for AI projects. The study projects that around 21 percent of the total wastage in AI projects in 2023 can be saved with effective project management practices. It also discovered that 76 percent of organizations use their own customized methodologies for DS/ AI projects.

Manage South Asia spoke to two senior leaders on their experiences of managing DS/AI projects.

Ruma Mukherjee is the technology leader for emerging technologies at Unisys, India. She is passionate about data analytics and has expertise in a wide range of qualitative and quantitative techniques in the areas of statistics, big data, Internet of Things (IoT), Natural Language Processing (NLP), Machine Learning (ML) and cloud-native computing.

Please tell us about the DS/AI projects that you are working on.
I’m working on establishing a culture of data-driven decision and digital transformation in many of the products across our organization. Most of my projects deal in making the ML models work in real production environments. These could be conversational AI or chatbots, AI in operations, real-time data analytics, analytics for Internet of Things and providing enhanced customer experiences through AI/ML-driven solutions. I drive the design and delivery of AI/ ML solutions – from data ingestion, data processing, model building and the deployment of models in production. I work with NLP tools and techniques, forecasting analytics, statistical and machine learning algorithms and big data.

What are the biggest challenges that you face in these projects?
 There are three major challenges:
1. Unrealistic expectations - Data scientists and developers struggle while experimenting with the available data and fitting it in the right algorithm to get an acceptable result. They believe that AI solutions are probabilistic and not deterministic. However, customers expect magic and believe their problems will be solved with 100 percent accuracy.

2. Immature domain - The traditional practices of product delivery that follow an agile or waterfall model fail in AI. The understanding of analytical techniques in companies is limited. Thanks to the uncertainties and low returns on investment, companies do not know how to meet the acceptance criteria of stories in agile.

3. Poor definition – There is no clear outline on the business requirements, the metrics to success, how to collaborate with the data engineering team, the strategies for storage and maintenance of data, or the right skillsets or structure of a data science team.

What are your recommendations for AI professionals?
1. Work on skill building and knowledge transfer across the organization where everyone understands the maturity lifecycle of a DS project.
2. Avoid force-fitting traditional development processes, tools and practices in AI-driven solution development.
3. Define a data management solution and rediscover the data collection strategies to get ‘appropriate data that works.’
4. Set realistic expectations and improve it over time with feedback and continuous improvement.
5. Start with an augmented solution and define a timeline of maturity for a complete AI-based solution.

How can organizations or professionals improve their capabilities to manage AI projects?
In today’s world, it is very important to stay relevant and continuously keep adding on to one’s skills. For that, I recommend the following:
1. Take professional help – courses, certifications and other such self-improvement programs.
2. Work with teams, network to understand what other companies are doing and contribute your knowledge to the AI community.
3. Establish a structured approach and best practice guidelines that address AI specific issues of uncertainty, data nuances, lack of skills and more.
4. Embrace AI in its infancy and help it grow to create that ‘magic’. Rome was not built in a day – understand the relevance of this idiom in AI and act.

---

Gopalan Oppiliappan, head - AI Center of Excellence, Intel India, is an AI thought leader who has been actively partnering with PMI and other leading organizations such as NASSCOM, Confederation of Indian Industries and Niti Aayog in accelerating AI adoption in India.

Please tell us about the DS/AI projects that you are working on.
I head the AI Center of Excellence (CoE) in Intel India. Through the CoE, we are tackling problems related to supply chain, product development and product validation with classical AI algorithms and deep learning techniques.

What are the biggest challenges that you face in these projects?
The biggest challenge I have seen is the lack of labelled data. Though we see data all around us, the lack of labelled data is a major bottleneck in training AI models. The second challenge is the lack of instrumentation of processes to collect logs of applications or processes.

Hence, it is pretty difficult to ‘sense’ what is happening in a system and to understand the ‘state’ of a system at any given moment. Without a clear idea of the state of the system, an intelligent decision cannot be made, predicted or recommended. We need to postpone the creation of AI use cases until adequate data collection mechanisms are put in place and the right data elements are collected over a meaningful period of time.

What are some of your learnings?
One of my major learnings is that the ‘mortality rate’ of AI ideas is close to 80 percent. That is, if we pursue 10 ideas, only two will get deployed. Given this grim scenario, we have to be super aggressive in communicating the status of AI use cases to our stakeholders and customers. We must manage their expectations upfront that miracles are not going to happen and failure rates are going to be high.

As a CoE, we also have to cast our net wide to generate a multitude of opportunities and sieve them systematically, so that the right ideas are picked for a proof-of-concept, evaluation and eventual deployment.

A clear entry-testvalidation- exit criteria is important to define and implement ideas in each stage of an AI lifecycle, so that the outcome at each stage is well communicated and expectations are aligned with stakeholders and customers. Every organization must develop this stage-gate approach and a ‘go/no-go’ decision criteria at every intermittent stage of an AI project.

This will help manage the expectations well within an organization. The syntax of the AI project stages is an interesting area for which we must evolve a standard. PMI has a great role to play here to evolve this common syntax of an AI lifecycle.

How can organizations or professionals improve their capabilities to manage AI projects?
Organizations have to invest now in developing an AI mindset across the entire hierarchy, right from engineers to CXOs. Only then will they have the ability to identify AI opportunities across a wide range of business problems. The second requirement is that program and project managers also need to understand the basics of AI algorithms and how to interpret algorithms, so that they can drive AI teams toward the right trade-offs and convert AI ideas to value with their stakeholders and customers. AI is not just for data scientists alone. It is important to develop an AI thinking across the organization. This will make an AI transformation much easier and faster in any organization.

Read the PMI-NASSCOM Playbook for Project Management in Data Science and Artificial Intelligence Projects