On the evening and morning of 6/27 and 6/28 Lumen conducted maintenance on all USDA One Network locations for every agency under the USDA. The goal of this maintenance was to eliminate a bug in the Cisco code that is running on all Lumen USDA One Network routers. However, this initial window failed. The failure caused outages across the entire country and affected emergency dispatching across the entire U.S. Forest Service network to include all 52 offices on the USDA One Network in U.S Forest Service Region 2. USDA Enterprise Network Services Duty Officer and Lumen had a troubleshooting call Saturday at around 0730 Central time to rollback the changes.
A follow-on Customer Service Notification stated the issue was resolved at 0930 Central Time. USDA Enterprise Network Services and Lumen then agreed to do a 12-hour maintenance window the night of 6/28 from 2100 Eastern time to 0900 Eastern Time and stating there may be "brief" interruptions. This was clearly not the case, and a full resolution was not submitted until 6/30 at 2330 Eastern Time.
Furthermore, a majority of the USDA One Network, U.S Forest Service locations in Forest Service Region 2, and even more specifically, every office on the Arapahoe and Roosevelt National Forest and including the following emergency service locations: Northern Colorado Interagency Dispatch Center, Craig Interagency Dispatch Center, Colorado Springs Air Tanker Base, Durango Air Tanker Base, and the Grand Junction Air Center, went down again on 7/1 at around 2025 Central time and were down for about 15 minutes. This does not include any other office that may have been assisting in any fire operations that were being conducted in Region 2. A total of 37 offices in Region 2 experienced the service drop on 7/1.
Although the maintenance was needed, not being rolled out in proper increments and it is unknown if the U.S Forest Service received any alerts that the maintenance was going to occur, all compounded to create unnecessary risks that could have been easily avoided and/or allowed the U.S Forest Service to implement a contingency plan/COOP for their emergency communications for the weekend.
This will be passed up through USDA leadership noting that this is not acceptable and puts not only U.S Forest Service personnel, Bureau of Land Management personnel, any other U.S agency assisting in these efforts, any contracted wildland firefighting pilots, and firefighters, but also the public at unnecessary risks of injury or death.
Although traditional radio communications are utilized, Radio over Internet Protocol (RoIP) is a part of the backbone for radio communications and any network outage, especially as widespread as this one, becomes a great concern.
|
