While data science adoption has been growing rapidly, analytics team processes and the models we use to evaluate them are still not clearly defined. In this article, we show the drawback of maturity models, and suggest an evaluation framework for data science teams based on the agile methodology.
What are data science maturity models?
Maturity models measure an organization’s ability for continuous improvement in a particular discipline. In data science, they focus on how well organizations can collect, process and analyze data and generate predictive models for making useful decisions. These models typically explore data infrastructure, data management, leadership governance and involvement, analytics performance and impact of the models.
While these models are useful for understanding an organization’s overall maturity in the data lifecycle, they are not useful for evaluating a data science team’s processes. Maturity is an important factor but doesn’t guarantee a well-defined lifecycle. However, they provide a good framework to build an evaluation tool for data team processes.
What is a good data science team process?
To deliver valuable insights, teams need to implement a repeatable process that is effective for analytics and data science projects. Building this lifecycle and aligning teams and stakeholders is the best way to deliver consistent value. We’ve found that the agile approach works well for data science teams and offers several advantages:
- Encourages small iterations where small failures are accepted
- Allows stakeholders to share feedback after each iteration
- Creates a prioritized list of project questions to answer
- Ensures stakeholders agree on this prioritized list of questions
- Aligns the entire team on project deliverables and timelines
How to evaluate data science team processes?
We have compiled 5 questions to evaluate a team’s data science process and share some best practices for each stage.
1. Life Cycle Usage
Q: Does the team have a fully defined lifecycle that is used by all team members and stakeholders?
Life Cycle Best Practices:
- Shared mental models are well defined and used.
- Lifecycle covers quality assurance (ex. bias, validation).
- Lifecycle includes data operations (ex. MLOps).
- Lifecycle is understood by extended team (ex. stakeholders).
2. Agile Coordination
Q. Does the team deliver value incrementally, and get feedback from stakeholders on each iteration?
Agile Coordination Best Practices:
- Have prioritized list of questions to answer (where each question is small).
- Each iteration has analysis & feedback and helps reduce project ambiguities.
- Stakeholders help prioritize iterations (choose tasks or questions to answer).
- Project goals and timelines are understood by team members and stakeholders.
3. Integrated Team Process Coordination
Q. Does the team have an integrated team process?
Team Integration Best Practices:
- The team process is fully integrated with the data science lifecycle.
- The team has a defined process to collaborate with other teams.
4. Team Metrics
Q. Does the team have defined metrics to measure project success?
Metric Best Practices:
- Define business metrics so everyone knows how success is measured.
- Track and optimize metrics based on team efficiency and effectiveness.
5. Process Improvement
Q. Does the team have an effective ongoing process improvement effort?
Process Improvement Best Practices:
- Regularly review team processes with action items for improvement.
- Regularly review data tools with action items for improvement / new tools.
How Agile is your AI team?
This article is based on Jeffrey Saltz’ experience with helping teams take advantage of emerging technologies and data analytics to deliver innovative business solutions.
