Agent Assist
Agent Assist is an innovative tool designed to provide real-time support and guidance to frontline agents in Customer Care with customer calls, leveraging cutting-edge AI technologies such as Natural Language Processing (NLP), Machine Learning (ML), and speech and text analytics.
My Role in a nutshell
As a Senior Product Manager, I led the Agent Assist initiative from inception to launch. This involved close collaboration with five core teams comprising over 30 individuals on a daily basis, ensuring communication and alignment with all parties involved, including senior management and the project sponsor.
Key Activities
- Conducted user research to understand challenges and pain points.
- Identified benefits and defined the value proposition.
- Defined product vision, strategy, and roadmap.
- Conducted detailed analysis of customer journeys for the selected call drivers.
- Defined requirements, MVP scope, priorities, KPIs and success measures.
- Evaluated development options, capabilities, and resources.
- Conducted market research and selected the most suitable 3rd party solution.
- Led product design and usability tests with the design team and Customer Care agents.
- Led product development with inhouse and 3rd party provider’s engineering teams to develop the AI-driven solution
- Facilitated technical conversations with internal engineering teams and with tech partners. Supported the team to manage and resolve dependencies, mitigate risks, negotiated with business stakeholders and made trade-off decisions.
- Monitored UAT activities, defect triaging and bug fixing. Acted as the arbitrator to resolve conflicts between internal and external teams.
- Supported preparation of training materials for agents and user training.
- Launched the MVP and collected agent feedback and data insights.
- Worked together with the model office team, IT support, delivery management and change management teams to coordinate Production testing.
- Evaluated qualitative and quantitative insights and identified areas of improvement.
- Planned for MVP scale and continuous improvement.
- Communicated project updates, including risks, assumptions, issues, and dependencies, to stakeholders in weekly meetings, ensuring complete transparency and alignment.
Benefits Expected in the Scaled Solution
Quantitative benefits included:
- Improved customer satisfaction (CSAT)
- Increase Net Promoter Score (NPS)
- Increased full-time equivalent (FTE) to customer ratio
- Reduced Average Handling Time (AHT)
- Increased first contact resolution (FCR)
- Increased Customer Retention Rate (CRR)
- Reduce the number of active complaints
- Reduce the number of new complaints raised
Qualitative benefits included:
- Empowering agents
- Increasing agent efficiency
- Reducing the risk of attrition
- Preventing complaint cases
- Receiving fewer repeat contacts from customers
- Improving work item classification and routing
The Problem - Issues and Challenges in Customer Care
Customer Care agents at OVO Energy faced significant challenges in efficiently supporting customer inquiries due to the need to navigate multiple systems. This led to operational inefficiencies and prolonged call durations.
Identifying customer complaints across various communication channels was also a struggle, negatively impacting response times, resolution quality, and overall customer satisfaction.
Agents were expected to maintain comprehensive knowledge across a wide range of topics, which was overwhelming. Additionally, finding relevant knowledge quickly from disparate sources to support customer interactions was challenging, resulting in extended call times and potential frustration for both agents and customers.
User Research
Remote sessions were conducted using video conferencing tools as well as in-person sessions to listen to live customer service calls to understand customer as well as agent needs and pain points, and how agents interact with various applications during their calls to support customers in real-time. The applications focused during the research were Salesforce Service Cloud CRM including the Knowledge Base, Google suite,
Conducted post-call interviews to gather in-depth feedback on agent experiences and challenges faced during the calls.
Solicited feedback through follow up surveys and direct communication from agents and business leaders to obtain a comprehensive understanding of the issues from both operational and strategic perspectives.
By employing these methods, it was possible to collect rich, actionable insights that informed the design and development of solutions aimed at enhancing the agent experience and overall efficiency.
High-level payee verification journey
Minimum Viable Product (MVP)
Product Vision
Enhancing customer satisfaction by significantly improving the agent experience and empowering agents to manage a diverse range of customer queries efficiently using cutting-edge technology.
Product Strategy
- Enhance agent experience: Streamline interactions by providing a unified interface that allows agents to manage all aspects of customer queries seamlessly.
- Empower agent: Equip agents to better understand and resolve customer queries, fostering trust and satisfaction.
- Drive efficiencies: Boost operational efficiency to reduce call duration, enhance work item classification, and ensure accurate routing for faster resolution.
Value proposition
Boost contact centre efficiency and customer satisfaction with real-time support and actionable insights.
Market Research
Due to the unavailability of internal tech resources and the low accuracy of data models developed by the internal Data Science team, it was decided to partner with a reputable solution provider to implement the solution.
I spearheaded comprehensive market research, evaluating Agent Assist solutions from leading providers to find a solution that would meet OVO’s business requirements and effectively address the identified issues.
Agent Assist solutions from four top providers were assessed.
Feasibility analysis
Vendor #2 was selected as the optimal solution due to its cost-effectiveness and reduced dependency on internal technical resources. This choice enables an immediate start and accelerates the speed of delivery, allowing OVO Customer Care to achieve desired outcomes more swiftly than leading competitors.
KPIs & Metrics
Following metrics were chosen as leading and lagging metrics extracted from multiple sources such as Looker, Tableau and regular reporting channels to evaluate product success of the MVP and scaled solution.
A new dashboard was set up using Mixpanel to track and report data insights related to the adoption of the Agent Assist application by the agents. This dashboard provides comprehensive metrics across several key areas to ensure detailed monitoring and analysis of agent interactions with the application.
This detailed tracking allowed for a comprehensive understanding of how the Agent Assist application is being utilised, providing valuable insights into its effectiveness and areas for improvement.
OKRs for the scaled solution
Following metric-based targets were expected to be realised with the fully-fledged solution within the first 6 months.
- 10% reduction of Average Handling Time (AHT)
- Customer Satisfaction Score (CSAT) > 65
- Full Time Employees (FTE) reduction of 75
Product canvas
Approach
Lean Experiment Approach was selected as the best approach to set and validate hypotheses, and prove the new technology can solve the intended problems for OVO Customer Care.
A minimum viable product (MVP) was created based on a set of hypotheses and validated with agents in real-time calls with customers.
Once proven successful, it’s planned to iterate based on value the solution delivers, measured by the KPIs and agent feedback.
Minimum Viable Product (MVP)
MVP scope
The MVP focused on
- Contact channel: telephony/voice
- Call driver: top five sub call drivers within Billing
- Team: Priority Service Register (PSR) team in Pay Monthly (PAYM).
Goal
The objective of the Agent Assist solution was to identify the extent to which AHT could be reduced using the Agent Assist features. The MVP solution included UI/UX designs tailored to meet agent needs, extensive usability testing, and close collaboration with OVO’s engineering teams and the solution provider.
High-level requirements
- Display real-time transcription of the call to the agent.
- Recognise call Intent based on customer utterances.
- Extract and display relevant predefined entities in real-time during the call.
- Automatically generate and display a summary post-call.
- Identify expressions of customer dissatisfaction and prompt the agent.
- Recognise customer vulnerability and alert the agent.
- Offer real-time guidance to the agent based on the call context.
- Suggest relevant knowledge articles to agents during the call.
High-level requirements
Sample user stories
Below is a sample of user stories with acceptance criteria created in the JIRA backlog, designed to guide the engineering teams through the implementation phase:
User story #1
User story #2
Hypothesis
- Providing the textual transcription of the live voice conversation will reduce the AHT (talk time) by 5% and improve customer satisfaction
- Auto-generating call summaries at the end of the call will reduce the AHT (talk time) by 5%
- Auto-generating call summaries at the end of the call will reduce AHT (wrap time) by 5%
- Prompting knowledge articles during a call will reduce AHT (talk time and hold time) by 5%
- Prompting knowledge articles during a call will increase customer satisfaction (CSAT) over 65
- Prompting knowledge articles during a call will reduce complaints rate by 3%
Measurement of success
The success of the Agent Assist solution hinged on demonstrating the accuracy of following:
- Intent recognition accuracy > 80%
- Agent guidance (knowledge and guidance prompts + entity recognition) accuracy > 80%
- Call summary accuracy > 80%.
Project plan
Solution
User experience (UX) and user interface (UI) were developed to meet the needs of OVO agents and align with overall product goals.
I led the design and engineering teams to prototype Agent Assist, creating a viable business and feasible technical solution and partnered with the design team to conduct user testing and collect feedback. from frontline agents, including the neurodiverse team, with a strong focus on the usability and accessibility of the solution.
Designs were refined based on feedback and prioritised considering the technical feasibility in alignment with the product strategy.
UI/UX design - Wireframes
Agents can interact with the Agent Assist solution through the menu located at the bottom right-hand side of the screen within Salesforce.
The content was organised into five frames for optimal usability and accessibility:
Frame #1: Displays live transcription of the conversation.
Transcription frame
Display real-time voice-to-text transcription highlighting negative and positive sentiment throughout the conversation.
Frame #2: Display call drivers identified and entities extracted.
Frame #3: Prompt relevant knowledge articles based on the live conversation.
Frame #4: Provides guidance prompts based on the real-time conversion.
Frame #5: Generate and display a summary of the call at the end of the call.
Call summary frame
Display a summary of the call with the ability to copy the auto-generated content enabling the agent to paste it into Salesforce memo and update.
Solution and implementation
I collaborated closely with the model office team and the Data Science team of the solution provider to fine-tune models, ensuring they effectively incorporated industry-specific terminologies. This partnership significantly enhanced sentiment recognition, identification of customer vulnerabilities, and extraction of relevant entities and call drivers during live calls.
I spearheaded the customisation and implementation efforts by leading internal and external engineering teams to tailor the third-party solution according to the final designs, aligning it with business requirements to boost product adoption. This included replacing the canvas-style UI with a streamlined single-screen solution integrated within Salesforce CRM, eliminating the need for application switching and ensuring unobstructed and easily accessible customer account content during the calls.
Tools and technologies
- Project management – Monday.com, JIRA Workplace Management
- Roadmap: JIRA Workplace Management
- Cloud platform: AWS
- Transcription service: Azure
- CRM application: Salesforce Service Cloud
- Design and prototyping: Figma
- Survey and feedback: Google Forms
Pre-launch activities
The solution underwent rigorous testing in the User Acceptance Testing (UAT) environment by the model office team of senior agents.
The model office team conducted comprehensive user training sessions in preparation for production testing. Agents received hands-on training, gaining practical experience with the application’s capabilities. I oversaw these activities to ensure alignment with objectives and a thorough understanding of product knowledge, demonstrating how it can enhance agents’ ability to support customer queries effectively. This approach ensured that agents were well-prepared and confident in using the application in real-world scenarios.
A comprehensive rollout plan was created collaboratively to deploy and test the solution in the Production environment.
Agents selected for the trial were configured and granted access to the application within Salesforce using Single Sign-On (SSO) authentication, ensuring a seamless and secure login experience.
I collaborated with the IT Support team to secure the necessary access and permissions for the selected agents to use the application within the Production environment via Salesforce CRM. By implementing Single Sign-On (SSO) authentication, the application ensured a seamless and secure login experience for the agents.
Product launch
Fifteen frontline agents from PSR PAYM team trialed the solution over four days, handling 771 calls, with 96% successfully transcribed.
Throughout the evaluation period, metrics from the selected agents were meticulously tracked.
Qualitative feedback was collected from agents via an online form, submitted at the end of each call. This feedback captured their experiences and assessed the accuracy of the information presented by the application, providing valuable insights for continuous improvement.
Evaluation
I evaluated success at the end of the production testing period by considering both quantitative analytics and qualitative feedback from agents.
To measure operational impact, I tracked metrics such as Average Handling Time, Call Time, Work Time, and After Work Time of the agents selected during the testing period and compared with their previous performance without Agent Assist solution.
I evaluated qualitative feedback captured from agents through an online form submitted at the end of each call. This feedback captured their experiences and assessed the accuracy of the information presented by the application.
Results of the experiment
Call handling and transcription success:
- A total of 771 calls were serviced by the agents, out of which 745 calls were supported via the Agent Assist solution.
- Overall, 96% of the calls were successfully transcribed by the Agent Assist solution. This high success rate was slightly impacted by unexpected issues with the transcription service, which failed to transcribe a small percentage of calls. These issues were reported to the technical partner responsible for the IVR system for further investigation and resolution.
- Approximately 37% of the calls serviced by agents fell within the call drivers scoped for the MVP, ensuring targeted and relevant data collection.
Accuracy levels delivered by the models:
- Intent recognition: 83%
- Agent guidance (including knowledge and guidance prompts + entity recognition): 69%
- Call summary: 83%
Metrics evaluation:
Agent feedback:
Key challenges faced:
Several challenges were faced throughout the initiative.
Technological challenges
- Complexity of the solution increased by other applications it interacts with.
- Inaccuracies in transcription service and require post-transcription correction.
- Transcription service failed to transcribe voice conversation to text during peak times.
- Failure on the IVR system to scale and communicate with the Agent Assist application caused latency in transcribing voice-to-text (up to 40 seconds).
- User experience and adoption wass limited by Salesforce UI design and interaction as well as technical limitations by the Lightning platform.
- Technical limitation causing additional complexity to prompt call summary before the end of the call.
- Lack of availability of data delayed decision making.
- Limitations in how knowledge has been organised in the Salesforce knowledge base limited the knowledge prompt capability only to prompt articles, which are lengthy.
Cross-functional collaboration
- Lack of documentation related to operational processes and availability of SMEs required to explore detailed scenarios and processes related to call drivers have taken a considerably large amount of time.
- Challenging to work with over 30 individuals across five groups.
- Prioritising and scoping of areas, call drivers, features and capabilities was time consuming.
- Proving agent experience feedback at the end of every call extended the After Work Time (AWT) of the call extending the already lengthy Average Handling Time (AHT).
- Testing and measurement of accuracy was time consuming and involved a huge effort.
Lessons learnt
Scale and Optimisation
Rollout the MVP solution
Following the successful Production testing of the MVP solution, I led operations leaders and engineering teams to create a comprehensive rollout plan to accelerate the benefit realisation.
Phase 1: Initial rollout to PAYM PSR team
In the first phase, the solution is scheduled to roll out in stages to the entire PAYM PSR team. This phased approach was designed to manage the adoption process effectively and address any issues on a smaller scale before wider deployment.
Phase 2: Expansion to additional Customer Care teams
In the second phase, the solution was planned to be rolled out to more Customer Care teams, prioritised based on following factors:
- Average Handling Time (AHT): Teams with higher AHT were prioritised to maximise the impact of efficiency improvements.
- Full-Time Equivalents (FTE): Consideration of team sizes to manage resource allocation effectively.
- Number of Open Complaints: Focusing on areas with more complaints to enhance customer satisfaction quickly.
Scale and enhance
I collaborated with leaders from the operations and engineering teams to develop a strategic plan for implementing UI/UX enhancements, along with new features and capabilities, to continually improve the solution.
Key areas of focus included:
- Support for more call drivers: Expanding the solution to cover additional call drivers to address a broader range of customer queries.
- Digital contact channels: Incorporating support for email and web chat contact channels to provide a comprehensive multi channel solution.
These enhancements aimed to increase the solution’s robustness and flexibility, ensuring it meets evolving business needs and customer expectations.