Conquer Voice Service Announcement March 15, 2022

Good morning, Conquer Voice users. Our development team has identified an issue that can cause the Conquer Voice softphone to experience intermittent delays in loading and button clicks. They are working all hands on deck to resolve and anticipate having an update to address the underlying cause deployed later today. 

UPDATE 12:45pm Pacific Time: The Development team has completed testing the fix for this item which will be deployed at 1pm PT. While the update is running, users may experience the softphone reloading or "Network Error." This will resolve naturally and the softphone will reload normally. The update will take approximately 5 minutes to fully deploy.

UPDATE 1:20pm Pacific Time: This update has been rolled back as it did not completely resolve the underlying issue. Additional updates will be provided here as soon as they are available.

UPDATE 2:10pm Pacific Time: The Development team has completed the needed modifications to the update and are preparing to re-deploy it. The re-deployment is scheduled for 9pm Pacific Time tonight. Resources will be available at all times to ensure a smooth deployment and functionality restoration tomorrow. An RCA will also be posted tomorrow. 

ROOT CAUSE ANALYSIS: 

Starting March 14th at 7:58AM Pacific Time, our monitoring system detected deadlocked systems in our Websockets, which handle all messages passing through each open Conquer tab for the user. No changes were made to these systems in several months and they had been operating smoothly during that time. Engineering rebooted the systems and added extra logs to help identify future issues at 9:00PM PT.

On March 15th at 8:05AM PT, our monitoring system once again detected similar deadlocking patterns. From the enhanced logs, Conquer Engineering identified a specific scenario caused by a high volume of quickly repeated requests (users repeatedly opening and closing SFDC tabs with Conquer in rapid succession) could cause the main processing thread to lock, resulting in a timeout. After timeout, normal recovery process then allowed all queued requests to process. The resulting end user experience was periods of slow loading leading to freezing, followed by a return to normal behavior. This pattern repeated a few times and Conquer Engineering put in a patch at 1:00PM PT. The patch was rolled back at 1:09PM PT as some users experience heavy freezing during deployment and was reapplied at 9:00PM PT without issue.

Conquer Engineering has been carefully monitoring system performance and have confirmed a successful patch to this issue.