10/28/23:
Approximately 1730 – The power to the Boise Dispatch building failed, the computers and radios on the UPS systems stayed on with power. The emergency lighting for evacuation came on. During the outage the internet/network, WildCAD server, and phone systems became inoperable. The power was down about 2.5 minutes.
Approximately 1733 – The power came back up, every radio in the dispatch office began to receive static on every channel at max volume regardless of channels being muted or not. The Dispatcher present closed the Catalyst program on all radios and brought Catalyst back up with the same result, every channel receiving static at full volume. This process took around 10 mins.
Approximately 1745 – The Dispatcher then turned the server racks off one by one in the radio room and brought them back up one at a time. The system corrected itself on some channels. They were powered down again to fix the remaining channels. While the server racks were off the LAN switch for Catalyst was making a sound not heard before. This was then powered down for a minute before bringing it back up. Finally bringing up the servers in the racks one by one. This resolved all problems with two exceptions, BLM Main and FS SOA South. Both channels had red X’s and were not working. A message was left with the local BLM Radio Tech via cell phone. Phones and internet for the office were down for the rest of the day.
1748 - A call to the BLM help desk via cell phone to file a ticket for the issues with the radio, network, and phone lines was made. Reference ticket #1130430. The Dispatcher had to leave a message because it was after hours and on a weekend.
Approximately 1808 - There was an additional power outage that lasted for only 10 to 20 seconds. No additional problems with second outage.
10/29/23:
0914 – Contact was made with BLM Helpdesk.
0926 – BLM Duty Officer was notified, and they alerted the AFMO and BLM Boise District Manager. The local IT Tech. was unable to be reached.
0952 – Contact was made with BLM Helpdesk to clarify what building was affected.
1005 – The State Telecommunications Manager and District Manager worked to get phones and internet back up.
1051 – State Facility Operations Specialist was able to get the phones and internet back up, but BLM Main radio and FS SOA radio still down.
1200 – Another ticket was created to fix the BLM Main and FS SOA radios. Reference ticket #1130433.
10/30/23:
Approximately 0830 – A phone message was left with local BLM Radio Tech. and contact was made with local USFS Radio Tech. The USFS Radio Tech. was able to fix the BLM Main and FS SOA radios while on the phone with Catalyst. At this same time, the Shoshone Radio Tech. showed up to address the radio issues for ticket #1130433. It was determined that a new PCIE card was needed for the FS SOA radio.
During the day and a half that a combination of phone/internet/servers/radios were down, the Dispatch was supporting RX burn operations and dispatched a new IA fire. The issue occurred on a weekend and accessing IT/Radio help was not quick or easy. Our "go to" for accessing critical help is the Helpdesk process, but if no one picks up the phone on the weekend then we have little means to get things going in a forward momentum. |
