Latest Announcements

Around 11:30 AM on Tuesday, June 28th, 2022 Sonoran CAD and Sonoran Radio's main database began experiencing high hardware utilization which triggered internal warnings. Subsequently, Sonoran CAD and Sonoran Radio customers began experiencing outages of services around 12:30 PM due to increasing resource utilization and backlogged queries. Our team at the time was already investigating the cause and attempting to implement a solution. This outage affected Sonoran CAD users for the full duration of the outage period between 12:30 PM and roughly 2:10 PM, Sonoran Radio users were affected intermittently through the duration of the outage period. Outages continued for about 1 to 2 hours as our team worked through challenges and implemented various solutions to address the root cause. We were able to track down the issue to unoptimzations with our NPGSQL Library that is used to interact with our database infrastructure. These unoptimzations stem from poor automatic optimization support for our newly implemented database engine, Cockroachdb, by our existing NPGSQL library. Our team has implemented optimizations in our code to resolve the outage and has continued to optimize further to reduce the possibility for similar outages to happen in the future. Future versions of our database engine coming later this year will also add to optimization efforts and decrease the probability we will run into this issue again. Our team is also in the planning stages of scaling up the database infrastructure to include multiple load-balanced nodes. This will make the database infrastructure more resilient and less likely to be overloaded. As always, we appreciate your support and business as a Sonoran Software customer. We understand unexpected outages can be annoying and apologize for the inconvenience. Thank you for being understanding as we continue to better our infrastructure and software to provide you with the best software solutions possible.

What Happened - Timeline of events: 6/27/2022 Scheduled database migrations of Sonoran CAD from PostgreSQL to CockroachDB began. This migration was performed in an effort to scale up database infrastructure to a multi-node setup with redundancy and load balancing options not easily achieved with PostgreSQL. After about an hour our team was able to resolve post-migration bugs that were not found in development testing and get the new database in a stable state. High resource utilization was noted as CAD utilization was higher in the evening after the migration. Further optimizations were added the following day to help mitigate this high utilization that didn't cause outages. 6/28/2022 An outage was observed and a separate post-mortem announcement was posted regarding that incident. 6/29/2022 Another similar outage was observed and the team decided to revert to the prior database infrastructure on PostgreSQL. Two primary issues were discovered only at scale: 1. CockroachDB currently offers poor performance on the pg_catalog virtual tables. The NPGSQL library used to connect the backend makes a request to these virtual tables to retrieve all database table structures and metadata. While only ~20,000/80,000 tables were migrated so far, this poor performance caused an initial starting connection query that hangs for multiple minutes after a backend restart. 2. Sonoran CAD's fully customizable records have searchable fields stored in varchar (text) arrays. While CockroachDB does support indexing for varchar arrays, it is limited to exact and case-sensitive matches only. For queries searching for a partial plate, name, etc. and searches where case sensitivity should be ignored (lookups ran in the CAD) this resulted in a full table scan. With larger communities maintaining 200,000+ custom records each, these full table scans cause connections in the pool to hang for 5+ seconds. The connection pool was then quickly filled and locked up during peak times when dozens of intensive searches are being run simultaneously. The decision to revert: These two issues identified had no resolution with the latest version of CockroachDB. With a timeline unknown and likely months away, adding additional hardware to make up for the optimization was not a feasible long-term solution. Data has since been "rolled back" to pre-migration on Monday the 27th. All services have since resumed normal operation. Data Recovery - Communities Created/Lost between 6/27/2022 and 6/29/2022: If you have created a community during the roughly ~48 hour period of 6/27 to 6/29 and have lost it due to the roll-back, our support team can help you recover: -Community Information/Customization -Community Servers -Department Structures -Ten Codes -Penal Codes -Database Sync Configuration -Webhook Configuration -Address Customization -Tone Board Customization -Community Record Templates -Community Records -Unit Identifiers -Community Characters Our support team will have you create a new community, migrate the lost data to it, and have your members re-join. Reach out to support.sonoransoftware.com to generate a support ticket request. NOTE: This data recovery is available only to NEW communities created during 6/27 to 6/29 that were removed due to the roll-back. Account Credit for Lost Time If you had an active subscription during this ~48 hour period of 6/27-6/29, our support team will compensate communities with account credit for three days (~10%). Reach out to support.sonoransoftware.com to generate a support ticket request.

What Happened - Timeline of events: 09:17 UTC Internal monitoring detected a network outage on all services for about 3 minutes in duration. At this point, we have been investigating a small number of (3-4 minute long) outages over the last week. This outage was longer and pushed us to open a new urgent ticket with our datacenter team to investigate further as our systems all were showing healthy results upon inspection. This ticket was in addition to another lower priority ticket already pending with our datacenter. 21:51 UTC A minute-long outage was observed and staff was closely monitoring the network for further outages. 22:10 UTC A 10-minute long outage was observed and reported to our datacenter team, at this point we only had confirmations that the datacenter team was looking into the issue further. 22:28 UTC Another 10-minute long outage was observed. 22:47 through 23:45 UTC A series of 6 outage blips ranging in 1-2 minutes was observed during this period as our datacenter team investigated the issue and reported back the culprit of these outages. 23:54 UTC Our datacenter team reported at this point that they had implemented a fix for a faulty VC port on an aggregation switch located upstream from Sonoran's managed network. At this time we noticed a short 2-minute blip that was attributed to implementing a resolution. At this point, we observed a stable network. Augest 25th 01:37 UTC, 02:32 UTC and 02:39 UTC Our team observed short 1-2 minute network blips which later was confirmed by our datacenter team to be attributed to implementing a more permanent fix for the faulty aggregation switch. Determined Cause and Sonoran's Response Our datacenter team was able to track down the cause to a faulty aggregation switch's VC port. This is common in large network deployments (ie. datacenter environments) and was quickly identified the same day there were multiple short network outages. VC or Virtual Chassis ports are used in redundant environments to provide interconnects between aggregation network switches. An issue in the system caused network traffic to Sonoran's environment to "flip" paths and caused the network blips observed. This hardware is outside our managed network at Sonoran and is maintained by our datacenter. Sonoran's Systems Administration team was able to effectively track and report network outages as they happened, resulting in a quick response. We appreciate your understanding and patience as the issue was resolved. Sonoran Servers Customer SLA Credit: Sonoran Servers Customers are entitled to an SLA credit per our Service Level Agreement. Sonoran Software Systems LLC has determined due to the nature of the intermittent outages in close proximity to each other that the entire duration of the time between Aug. 24, 2022, 9:50:00 PM and Aug. 25, 2022, 2:39:00 AM UTC qualifies for this particular SLA credit. In order to calculate your particular service credit please review our Service Level Agreement, section 10. These credits are time-based and calculated in relation to the outage duration and your service monthly cost. Per our SLA, customers have 72 hours from this post-mortem announcement to claim their service credit via our Support Portal.

What Happened - Timeline of events: 11/22/2022 14:27 UTC Internal monitoring detected a network outage on all services, internal investigation was launched and determined that the outage was upstream at our data center service provider that supports the backbone networking before our traffic reaches the internet. A ticket was opened after 15 minutes with the service provider and a phone call was placed to follow up after not hearing back after 30 minutes. The service provider was experiencing outages with their internal tools and explained they will get back to us as soon as they can, most likely once much of the outage would be resolved. Our outage was explained to be "most likely" related to the central outage they were dealing with. 18:10 UTC A follow-up call was placed to our upstream provider requesting further updates as their internal tools were appearing to be recovering. At this time the representative was unable to provide an answer but promised a follow-up shortly via their ticket system. A response was received within 10 minutes of the call asking us to recheck on our end if services were still down. At this point, we reevaluated our internal networking hardware for possible issues but were unable to locate any issues that presented to be the cause, from our findings we were seeing internet connectivity working to our management VPN but domain name resolution was failing on outbound requests from our network appliances, server nodes were reachable via management interfaces but failed to communicate over public-facing networking. Our team reported back to the data center our findings and we awaited further reply. 19:00 UTC Investigations continued on our end periodically as we await a reply from our upstream provider. Eventually revisiting a noted uplink port that was noticed previously we found additional issues. Due to the upstream issues and limited connectivity to diagnose our internal network, we dismissed a down uplink between network appliances to be part of the upstream outage. At this time we determined the need to dispatch technicians to the data center and investigate further as efforts to resolve the down link remotely were unsuccessful. 19:50 UTC Sonoran technician arrived on-site at the data center and began to access the down link. 20:15 UTC Sonoran technician determined that uplink ports on a network switch were not activating and determined that there was no issue present with cabling and other involved network appliances. Troubleshooting continued including multiple hard reboots of the affected network switch. After an initial reboot of the network switch, all ports on the switch failed to come back online. All logging indicated no issue with any network ports and the software on the switch indicated it was in normal working order but all ports were disconnected physically, which was not the case. 23:45 UTC After many hours of troubleshooting and exploring other options to work around the affected switch, the Sonoran technician was able to bring back online the downlink ports on the affected switch. Uplink ports were unplugged at this time for testing and would not come online when plugged in. At this point, we discovered and began confirming that the addon card that houses the uplink ports was in fact dead and there was no hope of recovering it. Replacement and quick turnaround purchase options were explored before and after this discovery but all available vendors were unable to provide hardware that would resolve the situation on short notice. Our team began implementing backup redundancy plans to implement a workaround. 11/23/2022 01:39 UTC Sonoran technician successfully reconfigured network appliances in a temporary state to get all services back online. Determined Cause and Sonoran's Response Our upstream provider was able to track down the cause of the initial outage to aging hardware crashes which resulted in services going offline. This was mostly resolved within 4-5 hours as they implemented backup hardware to resolve the hardware failures on their end. Our internal network failed shortly after or in close proximity to the initial outage of our upstream provider. Causing much of the delay in response to the hardware issue within our control. Once an internal hardware failure was determined our team worked as quickly as possible to resolve the outage, unfortunately, the hardware failure was not quickly identified by the hardware's operating software. Sonoran's Systems Administration team was able after hours of exhausting possible solutions and troubleshooting to track down the issue to a failed switch addon card that was used for uplink connectivity to our internal routing infrastructure. Our team implemented a temporary fix to bring the network back online the same day and have since started the implementation of a permanent fix and hardening of our hardware redundancy plan to ensure internal network failover in the event of a hardware failure in the future. Sonoran Servers Customer SLA Credit: Sonoran Servers Customers are entitled to an SLA credit per our Service Level Agreement. In order to calculate your particular service credit please review our Service Level Agreement, section 10. These credits are time-based and calculated in relation to the outage duration and your service monthly cost. Per our SLA, customers have 72 hours from this post-mortem announcement to claim their service credit via our Support Portal. The total duration of the outage as reported by our monitoring was 11 hours and 12 minutes.

This morning we experienced a partial outage, affecting Sonoran Servers customers. This outage originated upstream with our DDoS mitigation vendor. Network connectivity was quickly restored resulting in a very small amount of downtime. What Happened - Timeline of events: 1/18/2023 17:44 UTC Internal monitoring detected a network outage on some services, internal investigation was launched and determined that the outage was upstream at our DDoS mitigation service provider that supports some of the backbone networking before our traffic reaches the internet and recieves all our ingress traffic for scrubbing before it reaches our internal network. 18:21 UTC A ticket was opened with the upstream provider and a manual redirect was put in place to temporarily get services connectivity restored. 1/19/2023 10:15 UTC Investigations with our upstream providers concluded and routes were redirected to our DDoS mitigation provider once more. Determined Cause and Sonoran's Response Through investigation with our vendors we were able to determine that the issue stemmed from some route "leaking". Meaning some of our ingress traffic was going to the wrong upstream provider. This was due to a IP announcement configuration and how network providers will choose one announcement over the other. Our upstream providers are placing additional Configurations in place to help prevent this in the future. Sonoran Servers Customer SLA Credit: Sonoran Servers Customers are entitled to an SLA credit per our Service Level Agreement. In order to calculate your particular service credit please review our Service Level Agreement, section 10. These credits are time-based and calculated in relation to the outage duration and your service monthly cost. Per our SLA, customers have 72 hours from this post-mortem announcement to claim their service credit via our Support Portal. The total duration of the outage as reported by our monitoring was 40 minutes.

This week we have experienced a period of network instability. This instability was caused by a malfunctioning switch in the switch cluster upstream operated by one of our network providers. Network instability caused lower upload throughput (traffic flow from our servers to clients) and dropped packets periodcially. Our upstream provider worked diligently to obtain and install replacement hardware to resolve the degraded network performance. What Happened - Timeline of events: 7/23/2023 20:18 UTC Internal monitoring noticed network blips where services were unable to be reached for very short periods. These were dismissed with a little investigation as hiccups not originating on our network. As monitoring continued to alert us of blips we began investigating deeper, initially thinking some sort of internal routing issues were causing the behavior, this was later found to not be happening. Speed tests taken showed low upload throughput and some packet loss was recorded with each test. At this time a ticket was entered with our upstream provider in reference to the issue we had discovered 20:22 UTC Upstream provider confirmed investigating the issue on their end. After this message some clarification went back and forth between our teams to give further details relating to our tests and provide additional test results upon request. 23:52 UTC Upstream provider informed escalation of issue to their senior network administrator. 7/24/2023 17:29 UTC Our team asked for update as we have not recieved past their message about escalating the issue. 17:32 UTC Upstream provider informed us they are in the process of upgrading a faulty member of their switch cluster. Our team went forward with network maintenance on our end to update internal routing IP addresses that would later be phased out by our provider. Due to network instability we were unable to quickly diagnose a misconfiguration with this cutover and maintenance outages were slightly prolonged. This lead to needing to extend the maintenance period to complete cutover of our hosted software products that were unaffected by our misconfiguration. 7/25/2023 21:38 UTC Upstream provider confimed hardware replacement has been recieved at their datacenter and their team is working on deploying the hardware in order to replace the faulty hardware causing the network instability. 7/26/2023 10:13 UTC Upstream provider informed us issue should be isolated and network instability should be resolved. This was later confirmed by our team during business hours and notification sent to our customers informing them of the incident resolution. Determined Cause and Sonoran's Response The cause was due to a faulty network switch upstream in our provider's network. Sonoran's team was able to identify the issue within the hour of onset, inform concerned parties and communicate to customers effectively while awaiting for resolution upstream. unfortunately much of this issue was out of our hands and we apologize for the inconvenience. Our management team has responded with increasing the period of the outage for the purpose of customers recieving a higher SLA credit. In the future our team will work to improve monitoring to more quickly detect these types of failures and work to implement plans to more quickly resolve network instability in the event of an upstream issue like this. We believe this should not happen and the Sonoran experience is better than this, we are devoted to ensure that the Sonoran experience is one of the best on the market and we will continue to strive to achieve and exceed that goal. Sonoran Servers Customer SLA Credit: Sonoran Servers Customers are entitled to an SLA credit per our Service Level Agreement. In order to calculate your particular service credit please review our Service Level Agreement, section 10. These credits are time-based and calculated in relation to the outage duration and your service monthly cost. Per our SLA, customers have 72 hours from this post-mortem announcement to claim their service credit via our Support Portal. The total duration of the outage time has been rounded up to the next whole day by our management team and is 72 hours in total.

Monitoring and user reports indicated there was a network outage originating from our hosted servers at our datacenter. This was later to be found to be caused by some faulty routing rules put in place by our DDoS mitigation vendor during a larger attack. When this rule was removed network access was restored. Thankfully due to our recently expanded failover strategies in place with our hosted software, Sonoran CAD/CMS/Radio/Support remained online and available. Sonoran Servers suffered the most from the outage causing all control panels and services to become inaccessible. What Happened - Timeline of events: 12/5/2023 18:49 UTC Internal monitoring noticed network outage on server hosting nodes and websites, triggering investigation from our systems administration team. 18:51 UTC Tickets were opened with our datacenter to aid in further investigation, at this point in time very little information was able to be uncovered and simply a network outage was confirmed of some kind. Access was limited but not completely cut off. 19:30 UTC Upon further investigation we were able to determine that only traffic coming through our DDoS mitigation filtering was affected by the outage indicating the issue is upstream at that vendor. We opened at ticket at this point with our DDoS mitigation vendor. 19:45 UTC Our DDoS mitigation vendor was able to locate the faulty routing rule and remove them, restoring network access and all services to full functionality. Our team has since worked with our vendor to ensure that this issue isn't the result of any other misconfigurations and shouldn't happen in the future. Determined Cause and Sonoran's Response The cause was due to a faulty network routing rule upstream in our vendor's network. Sonoran's team was able to identify the issue within the hour of onset, inform concerned parties and communicate to customers effectively while awaiting for resolution upstream. unfortunately much of this issue was out of our hands and we apologize for the inconvenience. Sonoran Servers Customer SLA Credit: Sonoran Servers Customers are entitled to an SLA credit per our Service Level Agreement. In order to calculate your particular service credit please review our Service Level Agreement, section 10. These credits are time-based and calculated in relation to the outage duration and your service monthly cost. Per our SLA, customers have 72 hours from this post-mortem announcement to claim their service credit via our Support Portal. The total duration of the outage time has been rounded up to be 1 hour in total downtime.

Monitoring and user reports indicated there was a network outage originating from our hosted servers at our datacenter. This was quickly verified by our direct provider to be the result of an HVAC related failure incident further upstream in the Chicago internet exchange Equinix CH1. What Happened - Timeline of events: 1/14/2024 23:33 UTC Internal monitoring noticed network outage on server hosting nodes and websites, triggering investigation from our systems administration team. 23:42 UTC Ticket was opened with our datacenter to aid in further investigation, at this point in time we were able to verify that this outage was a complete network outage of our Chicago datacenter location. 23:56 UTC Datacenter provider replied with an update indicating datacenter temeratures have exceeded safe operating ranges, confirming technicians are already on site and investigating further. 1/15/2024 00:14 UTC Sonoran technicians request an update from datacenter at point in time. 00:29 UTC Datacenter provider promises update around 30 minutes from this point. 01:08 and 01:32 UTC Sonoran technicians request an update from datacenter at two points in time. 01:58 UTC Datacenter provider confirms teams from upstream provider teams have been actively troubleshooting chiller plant failures while additional teams deploy additional temporary cooling equipment in order to lower tempratures into safe operating temperaturres. 02:06 UTC Sonoran technicians ask for clarification on percise location of incident to access readiness for network restoration efforts. 03:15 UTC Datacenter provider confirms issue is within Equinix CH1 and that the temperature has been able to be lowered to a 120F average. The facility engineers at this point in time were successful in getting 2 out of 6 chillers back online, needing to get an additional 2 online in order to bring the temperatures into safe operating range for network operations to be restored. 05:11 UTC Network operations resume and services start coming back online. 05:19 to 05:34 UTC A short network outage blip occurs as result of network reconfiguration after connectivity has come back online. At this time Sonoran technicians confirm all services have been restored and failed back over to normal operating conditions. 17:31 to 17:34 UTC Datacenter provider has confirmed at this point in time their technicians were swapping fiber connections to restablish redundant links for our upstream network connectivity. Determined Cause and Sonoran's Response The cause was due to a major HVAC related incident at an upstream provider in the Chicago internet exchange Equinix CH1. Sonoran's team was able to identify the issue within the hour of onset, inform concerned parties and communicate to customers effectively while awaiting for resolution upstream. unfortunately much of this issue was out of our hands and we apologize for the inconvenience. Sonoran Servers Customer SLA Credit: Sonoran Servers Customers are entitled to an SLA credit per our Service Level Agreement. In order to calculate your particular service credit please review our Service Level Agreement, section 10. These credits are time-based and calculated in relation to the outage duration and your service monthly cost. Per our SLA, customers have 72 hours from this post-mortem announcement to claim their service credit via our Support Portal. The total duration of the outage time has been rounded up to be 6 hours in total downtime.

Early morning before regular business hours an outage was detected by our external monitoring service. An investigation was launched as soon as staff took notice during business hours. Improvements are being made to our monitoring notifications to hopefully help speed along these response times. What Happened - Timeline of events: 5/1/2024 12:06 UTC Internal monitoring noticed an outage on VPS node ORD01.SonoranServers.com, notifications at this time were sent to our team. 17:59 UTC Staff were alerted to the missed outage notifications and investigation was launched into the issue. 18:29 UTC Staff were able to resolve a bootup issue that caused VPS node ORD01.SonoranServers.com to not automatically come back online after a internal hardware error that caused an unexpected reboot at the time of the outage. Determined Cause and Sonoran's Response The cause was due to RAM failure in VPS node ORD01.SonoranServers.com. This caused an unexpected system halt which was followed by a message blocking the system from completing a bootup after the halt. Sonoran staff were able to quickly diagnose the issue and resolve resulting behaviors to get services restored in a timely manner once alerted. Alert time was not ideal and we as a result are modifying our notification parameters to better alert responsible parties for handling such outages. Sonoran Servers Customer SLA Credit: Sonoran Servers Customers are entitled to an SLA credit per our Service Level Agreement. In order to calculate your particular service credit please review our Service Level Agreement, section 10. These credits are time-based and calculated in relation to the outage duration and your service monthly cost. Per our SLA, customers have 72 hours from this post-mortem announcement to claim their service credit via our Support Portal. The total duration of the outage time has been rounded up to be 7 hours in total downtime. Only affected customers may claim an SLA credit, to check your eligibity please check your VPS service information in the Sonoran Servers Customer Panel. You will see "Server: ORD01.SonoranServer.com" once navigating to your VPS controls if you are an affected customer.

In the afternoon hours our internal monitoring alerted us to an outage of all our services for Sonoran Servers, thankfully all other services have been scaled out to multiple datacenter locations so they were able to remain online and unaffected by the outage. Services quickly returned about 10 minutes after the initial outage was detected. Investigation showed that this was a power event occuring at the datacenter which caused servers to loose power momentarily and reboot shortly thereafter. What Happened - Timeline of events: 7/22/2024 18:43 UTC Internal monitoring noticed an outage on all nodes in our Chicago colocation, notifications at this time were sent to our team. 18:49 UTC Servers at this time started coming back up on their own, around this time Dedicated server customers should have come back online amounting to about 7-8 minutes of downtime, including the time to boot into the OS. 19:28 UTC Sonoran Staff were able to restore network services to VPS, Game hosting, and TS3 hosting services after automatic booting was unable to establish network connection automatically due to the non-graceful bootup. Most VPS services were restored some time before this point, but this was the official end of the outage period as recorded by our monitoring. Network connection for individual VPSes may have taken a few minutes in addition to establish connection and be available. Determined Cause and Sonoran's Response The cause was due to a power event at our datacenter that sadly was not in our control to mitigate or avoid. This caused all servers and networking equipment to undergo an unexpected power loss and reboot, following thereafter booting network services were unable to successfully come up on their own due to the network hardware not being fully ready to serve the downstream servers when requested. Sonoran staff were able to quickly diagnose issues and resolve resulting behaviors to get services restored in a timely manner once alerted. Sonoran Servers Customer SLA Credit: Sonoran Servers Customers are entitled to an SLA credit per our Service Level Agreement. In order to calculate your particular service credit please review our Service Level Agreement, section 10. These credits are time-based and calculated in relation to the outage duration and your service monthly cost. Per our SLA, customers have 72 hours from this post-mortem announcement to claim their service credit via our Support Portal. The total duration of the outage time has been rounded up to be 1 hour in total downtime.