◯ Menu
To further AI understanding and adoption in the insurance industry, Lazarus AI is producing a series of Insights designated “AI-II” which stands for Artificial Intelligence - Insights for Insurance. This Insight introduces a framework for conducting an effective AI POC (Proof of Concept). The next Insight in the series will augment this article with additional perspectives on cross-cutting concerns that impact all parts of a POC.
In this Insight we will be using the term POC (Proof of Concept) to describe the activities discussed. The intent is to take an AI solution, apply it to an insurer use case, determine business value, and decide if the solution can be moved to a full production environment. Some organizations may view this as a pilot as opposed to a POC. Regardless of insurer nomenclature, the intent is to prove out solution efficacy. Although there must be a spirit of experimentation, the ultimate goal is to generate value.
Executing an AI POC has several unique hurdles that differentiate it from other technology POCs. The framework provided by this Insight is composed of four phases that will allow the insurer to complete an effective AI POC. Like many frameworks, the distinctions between phases may blur, but having a logical outline with specific goals will optimize the process.
All points of view in this document are based on Lazarus AI’s daily engagement with the insurance industry. Although written with insurance in mind, the model will hold up for other industries with special value for those that are also highly regulated.
Preparation has two fundamental components: operational elements and cultural elements.
The operational elements include technical and process items such as preparation of data, collection of documents, and identification of associated business rules. Moving out of the Preparation Phase does not require having every single data element or document across the enterprise, but rather a sample that is sufficient to explore a specific use case. The size and scope of the sample must be representative enough for the organization to determine if the technology’s added value justifies moving it to production.
Due to the heavily regulated nature of insurance and lack of complete regulatory clarity, each insurance company needs to consider its own AI guardrails as part of the Preparation Phase. This topic will be picked up again during the Partnership Phase, but at this point the insurer should have initial perspectives ready to share with the technology partner.
Security is another key component of the Preparation Phase. Decisions such as data masking and security requirements for the POC should be ready to share with the technology partner in the next phase. The POC SOW and other vendor processes in the Partnership Phase address the administrative aspects of security, but a clear understanding of the security expectations should first be established in this phase. Solely depending on the administrative processes usually proves to be a poor approach.
Some organizations may return to the Preparation Phase because the insurer is unable to pass through this gate on the first try. For example, the readiness and availability of data or documents may not be what the insurer expected when the POC was approved and additional preparation is needed.
Some CEOs and Board members may be surprised to find that their data and documents are not AI-ready.
Some CEOs and Board members may be surprised to find that their data and documents are not AI-ready. This is a potential roadblock even in companies that have made significant investments in data transformation, digital transformation, AI readiness, and associated programs. Company leaders who expect the insurer to be fully ready for POC work may get an unpleasant surprise at this stage. This should not be viewed as a failure, but instead as a reality-based learning experience.
The insurer must also work towards proper organizational engagement in this phase. If the POC involves any integration, as most do, a basic time commitment has to be made before launching into it.
This topic brings us to the cultural component of the Preparation Phase. For this topic, internal fortitude within the POC team is especially important. Companies will have to be ready for uncomfortable truths such as the need to further refine a document set further, despite past investments.
Another cultural issue is that insurers naturally want to drive out risk and uncertainty. Rapidly evolving technologies may cause a risk-averse institution to delay or completely avoid attempting a POC. However, in the context of today’s fast-moving world waiting may prove to be the largest risk of all. A key aspect of cultural preparedness is helping the institution understand the balance between the potential of emerging technologies and the perceived risk of an AI POC.
As 2023 concludes, many insurers have completed POCs and others have successfully moved AI solutions to production and are already receiving value. However, a few insurers might not be ready to execute a POC because of cultural or operational considerations. In this case, a candid conversation between the technology partner and the insurer is necessary. Together they can identify and complete all tasks left in the Preparation Phase before moving forward within the framework.
When the Preparation Phase has been completed the next step is executing the partnership model. One part of this phase is completing an SOW. Insurers are skilled at executing traditional SOWs, so this Insight will focus on some of the more unique attributes of an AI POC.
For this Insight, it is assumed that a third-party technology provider will be a key player in the proof of concept. Choosing the right partner is critical because a wrong decision may lead to an inaccurate conclusion that the AI solution has failed; when it was a partner failure.
Even when a competent partner has been chosen for the POC, defining clear roles within the partnership should be done upfront. If this is the first AI POC for the insurer, there will likely be new roles to discuss.
Although the technology provider is a key player, the insurer will have the most important responsibilities in the partnership. Even an AI technology partner with extensive insurance experience will not know all of the details surrounding processes or product complexities. The insurer must also bring detailed knowledge of the use case value and what drives that value to the partnership.
When it comes to data, it is not solely about availability but execution as well. The insurer’s comfort with data handling during the POC is non-negotiable. However, excessive restrictions on data may cause the insurer to miss out on opportunities for added value. For example, if documents are so heavily redacted that AI technology can’t possibly be optimized. As opposed to unilateral direction by the insurer, data handling should be jointly discussed in the Partnership Phase. If the technology partner is not sufficiently sensitive to data handling, then the insurer has the wrong partner.
There will be some skills that insurer staff will refine during the POC. One common example is strengthening prompt engineering capabilities with help from the technology partner. To ensure the most effective POC, the insurer should take an active role and develop self-sufficiency, even with a new skill such as prompt engineering.
Acquiring new skills is just as important as the technical aspects of the POC. If the insurer does not want to take on the new responsibilities, then the technology partner will take charge or a third partner will have to be brought in. If the insurer decides to handle prompting, then the POC should establish resources and personnel for managing prompt engineering in production. These discussions will be further covered in the Partnership Phase.
As previously noted, the insurer will be the expert on processes and data flows. However, in the Partnership Phase the insurer should be open to discussions of process modification if the technology provider expresses a strong conviction. Using advanced technologies on a suboptimal process will not maximize the value of the POC. Therefore, process modification needs to be considered a fair topic of conversation. Otherwise, any conclusion that the AI tooling can not add value to a use case may be inaccurate.
Choosing the right partner is critical because a wrong decision may lead to an inaccurate conclusion that the AI solution has failed; when it was a partner failure.
That being said, process modification can be taken too far. The fundamental transformation of major business processes may prove too complex for a POC, so the technology provider and insurer must collaborate to reach an optimal arrangement.
KPI (Key Performance Indicator) definition for the POC will also be done during this phase. It may seem more intuitive to place KPI definition in the Preparation Phase, but the constantly evolving nature of AI means KPI definition is not a task that can be completed in isolation.
A partnership between the insurer and the technology provider is critical when defining success criteria. One of the most important contributions from the insurer is current, fact-based results on cost and process metrics. If these results are unavailable, it should be communicated early on that the data is incomplete. Occasionally, we have seen insurers set expectations that are far above current standards. For example, setting a KPI barrier of 99% success with a current process success rate of 70% means that the insurer could miss out on a lot of added value offered by the technology. If the insurer is not confident that they know the current success rate, it is better to acknowledge this than to set an unrealistic barrier. The same applies to transparency on costs.
Another key component of the partnership between the technology partner and the insurer is implementing proper “day 0” controls (i.e., controls set before production). A few technology vendors may veer away from these topics, especially during a POC. Lazarus AI believes that insurers and technology partners must address items such as explainability, model monitoring, and human-in-the-loop requirements before executing a POC. Technology providers that do not partner effectively with insurers on these topics are not appropriate partners for the insurance industry.
Given the lack of complete regulatory clarity and the fast pace of technological change, each insurer will implement different controls. The reality Legal and regulatory ambiguity should be discussed in the Partnership Phase, but need not dissuade an insurer from a POC. Discussing these issues now and then testing out the approach in the POC will allow the insurer to get real experience with these topics as opposed to dealing with them in abstraction.
After preparation, the execution phase may be viewed with anticipation as it is the core of any POC. The specifics of each POC will depend on the particular use case, data, and documents that the insurer has chosen. However, some tasks are essential to the Proof Phase of any POC. Beyond the core use case, the Proof Phase should include an analysis of the scalability of the technology. Even if the use case only applies to a subset of some process, the insurer should understand how a full production load will work.
Another key task is capturing enough data points to evaluate the realized value and total cost if the solution is implemented at scale. This is why the discussion of potential process modifications needs to occur earlier on. If during the Proof Phase, team members suspect that more value might have been realized with some minor process modifications, an opportunity has been missed.
A failed value hurdle should be examined as it may be a learning opportunity rather than a summary indication of an unsuccessful POC or partnership.
KPIs used to monitor progress should be reported to the team frequently during the POC. An ideal POC allows for a degree of insurer self-sufficiency, but the technology partner should not be left in the dark. Instead, the technology partner should receive KPI results throughout the Proof Phase, as well as in the Pass Judgment Phase.
Another key component of the Proof Phase is assessing the regulatory & compliance items mentioned earlier. For the sake of brevity, this Insight will just use bias, as an example, as it is a widely understood example among insurers working on AI. For many companies that are behind on AI, the misunderstanding of bias has long been a barrier to AI projects.
As opposed to debating bias in the abstract, the best practice is to execute a POC with the assumption that there will be bias. Then the team can work to remove that bias throughout the Proof Phase. Doing so will allow teams to actively take steps to counteract the problem before moving AI into production. If working with the right technology partner, outputs with minimized bias are a reasonable expectation during this phase. This will require focused effort upfront but will prove to be more efficient and scalable than trying to remove bias from humans.
The best practice is to execute a POC with the assumption that there will be bias. Then the team can work to remove that bias throughout the Proof Phase.
The Pass Judgment Phase also has two dimensions, the first focusing on value. The team should start by evaluating all KPIs captured during the Proof Phase, looking for potential value added if the insurer were to implement the technology at scale.
Even if the primary use case does not meet the team’s standards, other use cases uncovered during the POC may add substantial value. If this is the first POC, the insurer may start to ideate on additional use cases during the Proof Phase. A failed value hurdle should be examined as it may be a learning opportunity rather than a summary indication of an unsuccessful POC or partnership.
Before moving beyond the Pass Judgment Phase, the insurer and the technology partner should review any roles and execution details that were identified in the Preparation Phase. New roles should be evaluated in light of ongoing organizational support.
If a POC has positive results in the value dimension, the next component of the Passing Judgment Phase is examining the current regulatory environment. This will be an imprecise evaluation as regulatory environments are continuously evolving. However, it is still a key piece of the Passing Judgment Phase that must be completed before the insurer can move to production.
AI technologies are moving much faster than other insurance technologies. It is vital to maintain a spirit of exploration and intellectual engagement during an AI POC, especially when each use case has unique product details, processes, and data.
A true partnership between the insurer and the technology partner is essential. Due to this first delta of the speed of AI technology evolution, insurers cannot expect a product to be delivered and tested in isolation. A tight partnership with supportive roles and responsibilities is a necessary component of any AI POC.
The second large delta with AI POC is the aforementioned lack of clarity with the regulatory environment and legal environment. As noted, this lack of clarity may exist for some time. However, this should not prevent insurers from attempting an AI POC as most insurance POCs are unlikely to face regulatory issues in the near future. Still, this is a topic that every insurer must address before moving a successful POC any further.
This Insight leverages Lazarus AI’s experience in the insurance industry to present a simple framework for conducting an effective POC. Many insurers have successfully completed a POC and implemented AI technology in production. In the coming year, many more will. We at Lazarus AI are available to help you whether you are just starting to develop use cases or are ready to dive into a POC of your own.
Lazarus AI develops enterprise-grade foundation models for the insurance industry and beyond. Lazarus AI’s advanced APIs enable organizations to eliminate their processing bottlenecks and provide rapid time to value.
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Block quote
Ordered list
Unordered list
Bold text
Emphasis
Superscript
Subscript