This article originates as a result from a few discussions I had about Minimum Viable Products “MVPs” and especially regarding AI products. The discussions sparked some thoughts in my head, for example, what are the essential parts of an AI MVP? Is there even such a thing as minimum when it comes to Artificial intelligent products? This led me to do some research on the topic and write this article to be some kind of a guide for myself and hopefully to others. Happy reading!
The purpose of an MVP is to lower risk and cost when developing complex products, where there is a lot of unpredictability and uncertainty. The idea is to build and deploy a simple product that could be done within a relative short timeframe, and at the same time create user value. The MVP is then used to generate new learnings on which to develop the product further in coming iterations.
In this article when defining our MVP, we’re going to build the process on the ”Lean Startup” methodology that consists of three main stages. Build, measure and learn. The Lean startup methodology starts with ”learning” by understanding what problem needs to be solved and for who we are solving it. Once you have your hypothesis framing the problem, the MVP solution and a plan on how you go about to measure and learn from your MVP for future iterations. You can go about building the MVP, testing it and hopefully sooner rather than later, deploy it.
You learn by evaluating your product on the initial metrics you decided at the start of the first iteration and the received user feedback. Find how you can add value to your product in the next iteration and repeat. So, this is the idea anyways of the Lean Startup. However, what I was searching for was some kind of framework to guide and aid the team to stakeholder communication in the start up of an AI-MVP. I didn’t find any good complete article covering it, maybe because AI-Projects in general are hard to define in advance but I took a stab at it anyways.
Data driven products and data products
For example, Google Gmail is an online email service that in itself is a product that uses data but isn’t per se a data product. Within Gmail, the spam-filter that identifies and classifies spam and malicious mail is a data product. Netflix is the product and the function that recommends you movies to watch is the data product. Do you get the difference? What separates them is that there are products that use data to facilitate an end goal like Gmail and many others, and there are products whose primary objective is to use data to facilitate an end goal. The AI MVP and this article focuses on the latter, whose primary objective is to use data to facilitate an end goal. Since the AI MVP is a data product, we go with the assumption that there’s already a product the AI data product could be integrated into.
I can’t stress this enough, stakeholders must come first. It will be the stakeholders that give you trust and means to develop your products and they will be the ones holding you accountable if and likely when things go wrong. Before starting, it is very important to discuss and align your goals and expectations. There is a good idea to set this at the very beginning and align again when you have framed the challenge, before you start building the product. Except aligning with the expectations, you also need to figure out how the result will be used in the end. The last thing you want to do is to build something that will collect dust on a shelf. It’s better to build something easy that will be used than something complex that’s too hard to be useful.
Try to answer questions like:
- What are your expected outcomes from this development?
- In what way would the outcome be utilized?
Defining the problem needed to be solved?
Problems can come in many different shapes and sizes but non the less, you are creating an AI product because someone has some kind of a problem or challenge. That person or organization is in a current state trying to move to a desired state, where there might be higher revenues for the organization or a better state where there is less manual labour. Upon moving between the current and desired state, there will be frictions in the form of challenges that work towards or against getting to the desired state.
Defining what problem you are solving, for who and why is key for delivering a viable product with minimum features. The better defined problem, the more precise your product can be. Below are a few support questions to help you define the problem.
Who is experiencing the problem?
For who are you trying to solve the problem, and who cares about what your AI-product will do? Who has the ability to influence the problem once given the proper tool?
What is the technical competence to solve the problem, can the MVP be more raw in Its interface or do we need a high level explanation?
Why is it a problem?
Start with asking, why is the problem even a problem? Sometimes, what appears to be a problem can just be a result of a simple misunderstanding or misinterpretation. I tell this from experience where I have developed BI dashboards to answer a certain question for a certain user. Then, after an organisational restructuring, this tool has ended up with a different user who is trying to answer a different question with the same tool. What happened was that the new user deemed the dashboard/data faulty or broken, though it worked perfectly. The problem is not the system, it’s the interpretation that is wrong, and the tool needed re-adjusting to fit the new user needs.
- To put a problem in a larger abstract, you can repeat the question “why” five times. This will help you get a bigger picture and find where you can make an impact.
How is the problem experienced?
How does the problem manifest itself and how is it experienced by the user. The simplest way to answer this question is to just ask the user.
What is causing the problem?
What concrete and observable thing is causing the problem? Can you define the problem with its physical properties? This could also give you clues later on as you figure out what data could be collected. Sometimes, this can be hard to define but it can very often be defined to a specific tangible event.
When is the problem?
Can you find temporal patterns in the problem that could give you a better understanding of what you are trying to solve? Can you find a certain interval, is it increasing/ decreasing, is it constant? How does it look in the past and the present, can you estimate the future?
Where is the problem located?
Is the problem related to a certain location, geographical och spatial on a part. Like finding patterns in time, this could get you a better clue as on where you might expect to find the problem.
What kind of product is needed?
We need to consider for whom we are building the AI-MVP, and what knowhow or competence the user has for what the AI-product will do. A data product developed for internal use may not be as complex of a development that may be needed for a private user. The output in an internal used dashboard to display customer churn may be more straight forward than the integration needed to display route-alternatives in an navigations app for example. When creating a Minimum Viable Product, you should always try to make it as simple as possible.
To help you along the way, start with answering the following questions:
Are there already existing models that we could use when we create our AI-MVP?
Are there any existing models that could be re-adjusted? Many AI models have already been developed and they can be found online, so don’t reinvent the wheel in the first place.
Can a simpler AI model in the MVP be created from existing reasoning and statistics?
A simple form of an AI model is a decision tree that uses ”if” statements to trigger actions. These are simple in construct but can be huge and complicated. So, are there any existing reasoning and statistics for the MVP you are trying to create that could be created with a simpler decision tree?
What function is our product going to have? Will it sort items into predefined classes or predict a future value or recommend options to a user?
Sometimes AI can seem a bit like a magic tool that could do almost anything. Although it is pretty amazing as a tool, it is still made up of mathematical algorithms that produce values as outputs. Some examples could be:
- Find historical patterns
- Take action
- Decision Making
- Create: deep fakes, speech, text
What kind of AI-model do we need? Do we need a deep neural network or are we sufficient in this stage with only a decision tree?
We touched upon this earlier, there are many forms of algorithms and models to be used when creating your AI-MVP. The trick is to find the more simple one that can be created and still deliver a value. In the picture below, you can see that there are some to choose from.
What platform is needed for the AI-MVP?
There are more and more platforms and tools popping up every year for creating Machine Learning models. You can, with some basic knowledge, create an AI prototype without any coding skills. One example is Peltarion.com where you can build ML-models with drag and drop tools to later on create an app in Bubble.io, where you can implement your model without typing a line of code. When creating your prototype or model, explore the most simple tools available in order to see if they can be used to rapidly develop and test your prototypes.
How well could the AI-MVP be integrated in existing systems?
A good way for your AI-MVP to be used and create value from day zero is to be integrated in existing systems. Through APIs, plugins, ad-ons or simple e-mail updates. This also lowers the need for the users to create new routines or mental models for using the product. Remember that we are still focusing on the MINIMUM viable product. If you map out where the product could be integrated in an early stage, it will become easier to plan for integrations in coming iterations of the product.
- What systems or tools do the users we are developing for already use?
- How well could our AI-MVP be integrated in existing workflows?
The AI-MVP must exhibit domain knowledge.
To elaborate on the previous point, we not only need to consider how the product could be integrated in existing systems, but also be aware of existing workflows. How well does the product assimilate into the users routines? This has also to do with ensuring usability for the product, since new behaviors and habits are not easily developed if there is no need for it.
- What routines and habits does the user have when the problem manifests itself?
- How would the user interact with the AI?
Value at day Zero!
The idea is to improve the product over time with several iterations and increase the user value. That does not mean that the first iterations should produce no value, of course not! You have to consider that the first user should be able to experience a value from your product right from the first use. Before you have sufficient live data to support your product, you may want to incorporate other non-AI support such as rule based technology or a ”human-in-the-loop” to ensure confidence for the product and creating value from day Zero.
- What other non-AI-methods could support the AI-MVP in creating value from day Zero?
So far we have gone through the outlining of the problem and the function of our AI-MVP. Now, we have to consider the most vital part of any AI model or data product and that is the data. Without it, we have nothing.
What data assets do your primary models rely on for training?
Based on your framing and model functions in your first iteration, map out what data assets you need to develop your model.
Do you already have sufficient data to train a somewhat effective model? If no, where do you find it?
Do you already have the data needed to build a model or do you need to find it? If you need to find data, a good place to start could be https://datasetsearch.research.google.com/
There is a lot of existing data you could use to create your first model, but be aware of the usage license. In some cases, you may have to collect it.
How proprietary is the data used to train your models?
The term proprietary data is used to describe data that is specific for an individual or organization. Proprietary can be data that gives competitive advantages and data that won’t be available outside from the organization.
How much data integration, cleaning, and other activities remain to be performed on your data before it is useful for training?
Data is almost everything when it comes to creating ML models. When trying to develop a simple model fast, it is quite dependent on how much you need to curate the data in order to create your MVP.
Do you envision additional data becoming available in the future to improve your models?
Try to think in advance. Will the product in itself or other factors make new data available to further improve the model in the future.
You should also consider the five Vs of data: Volume, Velocity, Variety, Veracity and Value.
- Volume: This refers to the size of the data. How much data is needed for your model and how does it scale with increased usage and number of users?
- Velocity: At what rate are you collecting data?
- Variety: In what shape and form are you collecting your data. Is it structured, unstructured or semi-structured?
- Veracity: How accurate is your data and how trustworthy is your data? This will translate directly into how much you can trust your model.
- Value: Don’t just collect a bunch of data. Collect the data that you could use for solving the problem and improving the model.
Feedback, learning and measurement
A crucial step in the Lean Startup model is the learning stage. One of the main functions of your product is to gather insights for its next iteration. So, before deploying or even building your AI-MVP, you need to consider what learnings you need and how these measurement devices could be built in from start.
- What data do you need in order to improve the model in the next iteration?
- What data can you collect about the usability of the product?
- What quantitative feedback could you collect from the users?
- What is the metric on which you deem the model successful or not? How do you know what features to kill or improve over the next iteration?
- How could an A/B-test be devised to measure performance?
Cost of error
AI-models are based on probability and they are likely going to make mistakes, especially in the first iterations of the model. So, you need to consider the cost of error before creating your model. What is the worst that can happen in the case of a false positive or a false negative. For example, the cost of error for a self driving car can be quite high. It could maul pedestrians down and that is a big error, so to say. On the other hand, the cost for a streaming service to make a bad movie suggestion is not quite as bad.
The cost of error can also function as a guide to direct you in your development phase. The higher accuracy you need from your model, the more time you need to spend tweaking and developing it. Sometimes, an 80% accuracy will work just fine and any extra development time will just be an unnecessary cost. Sometimes, when the stakes are higher, you may need more than 99%.
If the cost of error is high and a high accuracy is hard to achieve in the first iteration, you could always introduce a human in the loop to intervene when things don’t go as planned.
To sum it up, always think from the perspective that AI algorithms will be wrong and what happens then?
Good luck and remember to keep it simple
By now, you should have a pretty good idea of what the purpose is with your AI-MVP. What problem it should solve, what value it should deliver and what features it needs to make it happen. Remember to focus on the MINIMUM and VIABLE part. Show value as fast as possible to gain trust from your stakeholders. AI development can be very complex. It is not uncommon for AI projects to be aborted in the development phase before being deployed. So, remember to keep it simple and when needed, go with the closest AI-ish-solution to start of with and iterate over time.
If you wan’t to discuss AI-projects, don’t hesitate to contact us!
Further reading and sources: