Measuring the Value of Your Generative AI Pilot Implementation

By
Venkat Tummalapalli

Venkat Tummalapalli is the VP of Customer Success & Services at Nesh.

These days, everyone knows they must adopt generative AI into their processes or risk getting left behind. But questions and frustrations linger — particularly for enterprise companies in complex industries that deal with complex unstructured data. At Nesh, we see this with our customers across the advanced manufacturing sector including chemical manufacturing.

One of the biggest challenges these companies encounter is the idea of “Just do AI”. 

Forced adoption of GenAI solutions without clear next steps or rushing into a generative AI implementation without a clear understanding of the value you are trying to gain will result in a failed experiment every time. Similarly, doing a bunch of generative AI pilots with no clear next steps or no champion will also fail. 

When initiating any generative AI pilot implementation, the best course of action is to begin by defining the value you need for the generative AI pilot to deliver. 

Six Leading Practices for Measuring the Value of Your GenAI Implementation

I’ve seen Generative AI pilots succeed and fail for a variety of reasons, but the ones that succeed have clear commonalities. Let’s dive into the top six leading practices for getting value out of your generative AI pilot or implementation.

1. Engage and Activate Leaders

In some cases, AI projects are mandated from the top down and exploratory teams evolve out of that request. In other cases, AI enthusiasts may see a problem they are passionate about solving with generative AI. However, when a project comes together, the first step is to identify your team lead, your implementation team, and any leaders who have influence over whether your project will or will not move forward.

Starting the project off on the right foot is critical for momentum. Find avenues for collaboration to engage and activate potential leaders so they are invested in the success of your project. Determining your use case is an ideal opportunity to engage and solicit feedback. 

2. Define Value Metrics

Once you have your use case, you can start mapping to the value you want out of your generative AI pilot. Establishing clear metrics that align with business objectives will help you make the decisions to move forward or not once a pilot is complete. You’ll also have an easier time gaining buy-in from leadership or decision-makers. 

A few examples of value metrics could include: 

  • Faster onboarding by for new employees (i.e. less time to first sale for new sales reps)
  • Reduced time spent on specific tasks (i.e. Technical services require less time to answer customer application questions.)
  • Completion of more high-value tasks (ie. Identifying the root cause of Non Conformance tickets in a highly regulated manufacturing environment 

3. Socialize the Value

Once you’ve identified the core team for the project and the leaders with influence, it’s time to level-set expectations across any team involved in the project. Doing this early gives you greater control over how a pilot or implementation progresses from the outset.

Be sure to communicate the potential impact of the pilot implementation to the different stakeholders you’ve identified and address any concerns or misconceptions before moving forward. 

4. Test and Compare Solutions

Finding the best solution for you often requires testing and piloting different solutions. When you can evaluate the value of multiple solutions side-by-side, this will give you more clarity on how you want to move forward at the end of your pilot. 

The key to an apples-to-apples comparison though, is to focus on a specific use case and tackle that use case in the solutions you test with. In the case of Nesh, this could be: 

(I’ll share more ideas for generative AI use cases that deliver value later on in this post.) 

Evaluate how different technologies approach solving your defined use case and problem. Whichever use case you choose, be sure to balance any experimentation with value creation. 

5. Avoid the In-House Trap

One of the biggest mistakes I see companies make at this stage is falling into the “Let’s build it all in-house” trap. There are likely individuals across your organization who have already started experimenting with how solutions like ChatGPT or Claude can help them accelerate productivity. 

The combined simplicity of these tools and their enormous value often lead to excitement and overconfidence that an in-house team can custom-build a solution for the entire company to leverage. 

While there could be strong reasons to build certain solutions inhouse, in most cases, it ends up being a mistake. Six months later… the company is back to the drawing board, having wasted time, money, and personnel on an unsuccessful GenAI project that delivered no value at the end of the day.

Avoid this time-consuming distraction. GenAI solution providers have dedicated their full resources and attention to solving real problems that deliver value. They have specialized expertise and resources to purposefully build a solution for specific users and use cases. 

At the end of the day, you’ll benefit most from a solution that has already solved the myriad of unforeseen problems that arise when building any GenAI solution — let alone one that delivers value.

6. Understand Organizational Risk Tolerance

Any generative AI project comes with inherent risks. Different teams and different industries can tolerate different levels of risk. For example, industries and use cases that require a high degree of accuracy (~100%) will struggle to find a use case that suits their needs. 

Today’s GenAI requires a certain level of human oversight to ensure accuracy. In the case of Nesh, we build safeguards to help improve the accuracy of information, but the outputs are only as good as the source material.

Make sure you assess your organization’s risk tolerance and acknowledge GenAI’s limitations. Once you’ve done this, you can ensure there are appropriate guardrails to keep any risk from your pilot and use case on track. Some Organizations conduct a formal Responsible-AI assessment to ensure the adequacy of safeguards. 

GenAI Use Cases that Deliver Value 

At this point, you may still be wondering how to define a use case that will deliver value. You’re not alone.

Most companies fail to move forward with GenAI because they don’t know how to identify the right use case. Forget a use case where they can derive substantial value that offers a transformative ROI. 

Build a Reliable Single-Source of Truth

Ensuring the accuracy, completeness, and readiness of chemical product data and knowledge is no easy task. However, when you consider the mission-critical processes that rely on an accurate corpus of information, the urgency and ROI quickly become apparent. 

Mastering your company’s knowledge spread across vast stores of unstructured data base is difficult in any industry. In the chemical domain, the challenge is heightened. Take, for example, a single agronomy handbook. Full mastery of this single data source is nearly impossible for most individuals. Layer in more and more data and the task becomes more and more complicated.

Similarly, maximizing how your team capitalizes on a subject matter expert's career worth of knowledge is another component that often falls by the wayside until it’s too late. One-off questions are inefficient and lost and repeated over time.

Combining your company’s unstructured data and expert knowledge into a GenAI-powered single source of truth enhances your team’s ability to serve as experts worthy of trust in critical situations, help customers meet their regulatory requirements, and leverage the right product positioning and consistent messaging in the field. 

Keep in mind that creating an AI-powered single source of truth can serve a variety of purposes. Identify a simple use case for your pilot and test team so you can focus on building the right corpus of documents and expert knowledge for the results you want. A clear understanding of how the solution will be leveraged will help you communicate value from your pilot.

Here are a couple of ideas to consider adapting to a narrow use case: 

Retain the Knowledge of Retiring Experts

Mitigating corporate knowledge loss is critical for most companies in the chemical industry as experts near retirement age. Capturing knowledge before the expertise walks out the door helps maintain business resilience.

In the case of Nesh, we use best practices for knowledge capture and recall combined with generative AI. We work with experts to capture their knowledge before they retire. We then layer this knowledge on top of a specific corpus of data to build a question-and-answer system that replicates the expertise users need. Users can query the Nesh platform and get answers to questions long after the expert retires.

Improve Product Velocity and Quality Control

Companies that manufacture a high volume of parts for complex machinery, may also have to deal with a high volume of Non Conformance tickets based on the rigidity of the underlying engineering designs required to conform with regulatory and quality control requirements. Sorting through vast amounts of data to identify high-priority tickets can be a challenge with no small impact on the bottom line. 

Generative AI can assist in prioritizing those non-conformance tickets and engineering issues to avoid rework and accelerate product velocity. Engineers are then able to analyze the issue, get to the root cause of the problem, and save the company time and money.

What do these use cases have in common? They are attached to clear business objectives with tangible results. 

Need more ideas? Check out our Fact Sheet: 6 Use Cases for Chemical Sales.

Sustaining Value of Your Generative AI Implementation into the Future

So you’ve followed the guidance outlined here and had a successful pilot. You’ve demonstrated value. You’ve selected a solution that meets your business needs.

And leadership is asking you, “What’s next?”  Here are some strategies to sustain momentum and drive value from your Generative AI implementation. 

Enhance User Stickiness

Users may have shown significant interest in your pilot use case but getting them to sustain their interest and channeling that into regular usage is your next priority. You’ll want to explore new use cases that align with the one that showed value in your pilot and build the solution into your existing workflows and workstreams. 

Figure out how to leverage technology integrations for a seamless user experience. 

Monitor and Quantify Daily Usage

Tracking user engagement and adoption rates will help you determine the sustained value of your implementation as you expand use cases and bring on new users or teams. Implement analytics to measure daily active users (e.g. ensure the quality of answers and responsiveness is aligned to user’s expectations). 

Demonstrate Top-line Value

The value of your GenAI implementation at any stage should always tie back to business priorities. As your implementation changes and grows, continue to emphasize how the AI use case connects to revenue growth (e.g. deal velocity, new sales hire ramp time). Take the time to continue to cultivate the support of your organization’s leaders or decision-makers by clearly communicating how the ROI ties to concrete business outcomes. 

Continuously Improve and Iterate

Stagnation and complacency will eat at your success over time. Regularly reassess and update value metrics — especially as new business priorities arise. Taking the time to adapt your AI solution based on user feedback and changing business needs can generate surprising dividends or results. 

Scale Your Success Across the Organization

Don’t stop now! Keep your champions activated and document your success stories. When it’s time to scale, ask yourself: 

  • How can GenAI improve other processes in the same business unit?
  • Are there other use cases this GenAI solution can tackle?
  • Are there other teams or business units that can use the same use case that proved successful in the pilot?

Generative AI is likely still an unknown entity to many in your organization. The time you’ve spent shepherding your GenAI implementation has made you an expert in the solution, use case, and pilot process. Be ready to share your experience and expertise as others turn to you for guidance.

Getting the Most Out of Your GenAI Pilot and Implementation

Knowing how to measure and communicate the value of your GenAI solution is key to its success. Start by setting clear goals and metrics that align with your business objectives. Whether you're aiming to retain expert knowledge, improve safety, or boost efficiency, make sure your AI efforts support these goals.

Remember, your efforts to derive value don’t stop at the end of your pilot or initial implementation. The real potential of generative AI starts after your pilot so keep your focus on continuous improvement, growing user adoption, new use cases, and scaling to new teams. Keep track of how often people use your AI solution and how it impacts your top and bottom line. This will help justify ongoing investment in AI technology.

The path to successfully adopting generative AI requires careful consideration beyond the desire to build in-house or rush undefined projects. Focus on smart implementation, learning from pilots, and expanding what works. 

A thoughtful approach will help your business become a leader, making the most of AI's potential and creating lasting value.

Want more? Download our ebook, The 2024 Guide to Generative AI for Technical Sales

Related Posts
Want Nesh Insights Delivered to Your Inbox?
By clicking Subscribe to our blog! you're confirming that you agree with our Terms and Conditions.
See how Nesh supercharges your sales cycle
Customer Experience
Technical Sales
Research & Development
Quality Assurance
Technical Services
Channel Partners
Business Development
Customer Experience
Technical Sales
Research & Development
Quality Assurance
Technical Services
Channel Partners
Business Development