Colleen Qiu is a former Head of Data @ Metromile with experience in AI, data science and analytics across various business domains at both Fortune 500 firms and high tech startups. Previously, she has worked at Albertsons, Tesla, PayPal, eBay, Chegg and others, in a variety of data-related roles.
She specializes in leveraging data science to solve business problems, and has wide experience leading analytics teams to support to business functions such as: A/B testing/experimental design, Marketing, Product, Customer Insight, Strategy, Risk Management and FP&A.
Read the full white paper here ➡️ Deliver AI Faster With The [2x3] Methodology
Challenges in AI Development:
The path of AI development is full of challenges. Success takes a combination of leadership, engineering, software tools and planning strategies. When a project fails to deliver customer or business value, it can have expensive consequences. I will outline a few common roadblocks during the AI development stage, and present the 2x3 model to overcome these challenges.
First, science often fails to comprehend business. AI teams are staffed with computer scientists, statisticians, mathematicians and others with technical backgrounds. While these engineers may be talented in their STEM field, they generally lack business domain knowledge. And when there are no bridges between the business teams and the AI scientists, it becomes difficult to build solutions that solve real-world problems.
When there are no bridges between the business teams and the AI scientists, it becomes difficult to build solutions that solve real-world problems.
Another issue is the turnaround time to convert scientific models into real-world solutions. There are usually no definitive answers in data science and AI solutions. Deciding when a solution’s accuracy is good enough becomes more art than science. Without using the right progress models, teams get lost in the pursuit of perfection and will fail to meet the deadlines.
Sometimes teams end up working on the wrong problem or solution for too long. This happens when data science teams fail to understand the business needs. This is why a short iterative cycle is helpful to catch early mistakes, without letting them accumulate and cost resources. Tracking progress with the right software tools helps to catch early mistakes.
2x3 Model for Delivering AI
To overcome these challenges, I suggest the 2x3 delivery model for AI projects. I employed this model while leading data science teams over the years to engage effectively with various business teams and ship out AI solutions. Its short iterative cycle enables consistent AI delivery to meet business needs, leading to broader AI adoption in an organization.
The lifecycle of an AI solution includes the following steps:
- Identifying and sizing a business opportunity
- Exploring and evaluating the status of data
- Researching various methodologies to find the best solution
- Gathering business feedback on the solution design
- Deploying solution to measure and evaluate impact
- Rolling out and continuously monitoring the solution
- Refreshing and updating existing solution
We map this to a 3-stage business engagement model: “Discuss”, “Build” and “Deliver”. Since we are allocating 2 weeks per stage, it takes 2x3 weeks for our solution to launch.
Discuss
First, teams should clearly define the business needs and evaluation metrics. Everyone involved must understand the business problem before brainstorming a solution, and they must agree on how success and impact will be measured after delivery.
Given a problem statement and available data, engineers can now propose solutions. Ideas need to be chosen from a business standpoint that justifies the upfront investment cost to build an AI solution. Teams should evaluate if it’s worthwhile pursuing the project before starting. Once agreed upon, leadership can allocate resources and design the roadmap for final delivery.
“Those of us who have built and shipped commercial systems know that there's a lot more to building the AI system than training the model. When I work with AI teams, I encourage teams to think about the entire life cycle of a machine learning project, which includes scoping the project, defining what's worth working on, and what projects to pursue." - Andrew Ng, Scale AI Transform keynote
Build
During this stage, data scientists start developing prototypes. Teams should leverage the power of scalable platforms. They allow for quick, iterative development of proof-of-concepts, with the potential to scale along with the solution once it’s delivered.
To achieve the two-week delivery cycle, it helps to anticipate common machine learning problems. Some industries rely on just a handful of ML algorithms (i.e. regression, time series, clustering, optimization). Anticipating in advance allows teams to work more efficiently and save time during the building stage.
Now is also a good time to demonstrate a solution to business users, and show how its impact will be measured. This will help understand if your solution is aligned with their needs and see if they would be comfortable using or implementing your idea.
Deliver
This is the critical phase where we ship and find out if our solution works. We’ll try to deploy the AI solution to a smaller scale and design experiments to validate its performance first. If that’s not possible, we can run a simulation or build a scenario analysis before releasing our solution.
Establishing a process for going to production is essential. This will schedule the team’s solution launch, engage the data engineering team for data needs, leverage the production support team for monitoring needs, and plan the downstream engineering team for system integrations. Monitor jobs, trigger alerts, and report business impacts once the solution is delivered.
After launch, when the model performance degrades or more data and newer techniques are available, the team can go back to refresh or upgrade their solution. Teams should plan out renewal cycles in this stage, so they are prepared when review time comes.
Adding value with [2x3]
The 2x3 AI delivery model is a simplified operating model to gain traction and show the value of AI solutions. This model is straightforward for business partners to follow. It helps the data science team deliver projects within a reasonable time frame. The model breaks large projects into smaller pieces, helping to expedite AI solution delivery and increase the team’s throughput.
