Hello, Conquer Voice users. We are receiving reports of Conquer Voice not fully loading and an inability to make or receive calls. Other areas of Conquer could also be impacted. Our development team has zeroed in on where the error is occurring and is currently working to resolve the issue. We will update this post as information becomes available.
UPDATE - 8/18, 4:10a PT: Affected users will see a screen indicating that Conquer Voice is loading, but loading will not complete. Development and engineering continue working toward a resolution.
UPDATE - 8/18, 4:52a PT: Our development team continues working internally and with 3rd party providers who are a part of our network stack to resolve the identified issue.
UPDATE - 8/18, 5:35a PT: Our team is intently engaged with the involved 3rd party provider to resolve the issue as well as preparing more drastic mitigation measures.
UPDATE - 8/18, 6:40a PT: Service has been restored. The dialer now loads and other effected areas are operational. We are escalating this with our 3rd party provider for an RCA. We will update with our own once it's available.
At 2:30 am PST 2021-08-18, Conquer support was made aware of an issue where several Conquer backend systems were unable to connect to the Conquer database. The Conquer team identified the problem as a core networking issue with its database hosting partner and immediately reached out to their escalation team for support. Conquer’s database hosting partner investigated the issue and tracked it down to unscheduled maintenance work that had been performed which caused the issue. The maintenance was then rolled back which lead to Conquer systems returning to working function. By 6:37am PST all services were confirmed restored.
Conquer’s provider has since confirm that the network maintenance was in fact the root cause of the outage experienced by their customers, including Conquer. By design, the Conquer backend database is redundant and hosted by said provider in 3 separate (triplicate redundant) servers. Through this event, we were advised by our provider that these servers, while redundant, are all supported by a single network segment, which was the target of the unscheduled network maintenance. Having a single threaded component does not provide service availability that is acceptable to Conquer. We are therefore planning a reconfiguration to completely move off of our current provider to ensure that the network design and redundancy meets our strict resiliency guidelines.
Ensuring the availability of the Conquer platform and the security of your data is of the utmost importance to the Conquer team. This maintenance activity allow us to continue to provide excellent service and will deliver an increased availability and resiliency to issues.
Conquer Engineering has built a migration plan to be deployed in the next few weeks. This migration will be regionally redundant and not tied to a single network segment. The cutover from the old service to the new service will occur in an announced service window after US business hours. We expect the cutover to be service impacting, but should take no more than 15 minutes to complete. No action will be required by customers to make use of the new configuration. More information will be shared as our plan is finalized. Thank you for your patience.