Good morning, Conquer Voice users. This morning, our developers observed some errors on our Campaigns service, and some users have submitted reports of slowness to load or serve the next record. Our developers are actively working to resolve the issue now. This post will be updated as new information becomes available.
UPDATE - 6/28, 8:01a PT:
The dev team is pushing some updates to Campaigns. All users on Campaigns will experience slowness for a few minutes while these roll out. We'll provide another update once those are complete.
UPDATE/RCA - 6/28, 11:25a PT:
At 6:15AM PT an issue was identified related to Campaign Schedules containing a high volume of Child Campaigns with no records which were all scheduled for use at the same time. Repeated attempts to serve records where none were available caused the service to become overloaded. This caused slowness in the UI for agents.
At that point, our internal health checks were past reasonable levels so the Campaign service was forcibly reset. In addition, Conquer Engineering team started to increase the server count. General service outside of Campaigns was restored to normal speeds.
At 8:30AM PT a temporary update was applied to allow Campaign service to return to normal operation as it rolled to all servers throughout the morning. We have increased server capacity by 5x and are actively monitoring system load and performance. Conquer Engineering is currently working on a permanent fix to allow for the Campaign Scheduling feature to be used with multiple empty campaigns at the same dialing times without impact to performance.