On 3/17/2024 the Arapahoe Roosevelt SO, Building E, located at 2150 Centre Avenue, Fort Collins, CO, 80526 was scheduled to have all 7 network switches upgraded per a Lifecycle Management Project. A Project funded by USDA-NRE (Natural Resources and Environment, Forest Service), executed by USDA-CEC (Client Experience Center). The scheduled project was slated to begin at 0930 MST. The site requested the upgrade to be pushed back to 1100 MST to ensure a COOP could be enacted to allow dispatch operations to continue. The radio tech for the Arapahoe Roosevelt National Forest, while assisting in the COOP, noticed links beginning to fail around the times of 0915-0930 MST, indicating the removal of cables from switches. The radio tech was able to reestablish communications using another port in a different switch, and the contracted tech proceeded to swap that one out, which should have been a quick swap, but took the contracted tech 1.5 hours to swap the two dispatch switches.
Furthermore, the contracted company that sends out the equipment to the site and adds the configurations to the devices, missed the configuration that allows the radio traffic to traverse over all the offices network switches. This caused further delays in radio operations that impacted fire operations as well. At no point during this process was our USDA-CEC network tech told by the contracted tech regarding the request for additional delays except for the holding for the COOP setup. The additional request for delays were not passed to the USDA-CEC network tech, this individual did not hear any communication between the contracted tech and the local contact.
Remediation steps will be added to the process documentation to prevent something like this from occurring in the future. Safety is the number one priority for USDA-CEC assisting with USDA-NRE. Any controls that can be made, will be made; to accommodate the end-users we assist daily within our service level agreement parameters.
For ALL USDA/Forest Service IT-related issues/outages:
The most efficient means of obtaining assistance with IT difficulties is by opening a ticket with the Forest Service Customer Help Desk (CHD) at 1-866-945-1354 (TTY: 1-800-877-8339). The user/customer should request a ticket be opened, reporting as specifically as possible WHAT system(s) or capabilities were affected, the date and time WHEN they were impacted, WHERE the impact occurred, and WHO can provide additional information that may be required. Please include the ticket number in ALL correspondence related to the issue/outage, including SAFENET if one is initiated.
|
