Missoula Interagency Dispatch has been experiencing intermittent radio issues. In addition, several dispatch centers across the country have experienced total outages of network, impacting their ability to communicate via phone or radio, and leaving them without the ability to use online programs. Now, specifically for Missoula - several base stations for our radios stopped working. We had mitigations in place and were using those, but then those also stopped working. The radio techs were advised and CIO/CEC help tickets were created. Without these radios, we have lost the ability to communicate with field going personnel (including fire and non-fire) for an entire zone and for half of a another zone. This is critical, as we have several active fires with several field going personnel to locations that do not have cell service, leaving radio as the main means of communication. These frequencies impacted are also used for flight following of DNRC aviation resources, as our flight following channels cannot reach for all of our zone. The back-up options in place do not work due to the type of issue we have at hand. Through the investigation of many, it appears to be an issue with Lumen Technologies - a new contractor in which as USDA offices are getting their network switched over to. While the situation is still ongoing, it is believe that Lumen has placed a security related firewall over the radio over IP addresses, thus rendering them unusable.
Below is the language being used at the higher tiers for LMR:
Folks, there is a bigger issue going on. I have opened a ticket with DISC/ENS. We are in a waiting game right now, waiting for them to get it fixed.
XXX stated the following to me just a few minutes ago.
“As for what's been going on. I just got done finding a weird issue with ISE. The load balancers flipped HA pairs last night, and come to find out the physical interface of the now primary F5 is taking errors, so pretty much anything relying on ISE has been having issues this AM.
But, if you're talking about multicast issues, I don't know but there sure seems to be something. I got pinged by a tech in R9 yesterday with those weird issues where I could ping the IP-224 but couldn't access the web page. He was saying one way audio. I opened a ticket w/ ENS and like 5 minutes later, everything started working. I don't think that it even got to Lumen yet, but it had been broken for almost a day and then was magically fixed. I got roped into the ISE issue when I was in the middle of looking into what you posted earlier today, but it looks like the same thing. I can ping the 4 devices you listed, but I can't get to the web pages for them. It's almost like a firewall issue or something, but I'm on Lumen as well and didn't think I went through any firewalls to get to the devices. I can open a ticket w/ Lumen to have them clear the SD-WAN (OMP) which fixed the issue the other day. But, I'm totally confused as to why it seems to be happening at more sites all the sudden.”
FYI: Another email from XXX to the Firewall group. Good afternoon,
We are having issues accessing devices at site NSC: SLLKMTAC
I am able to ping all equipment at the site, however, I am unable to SSH to my switch nor http/s access devices running those services. However, if I first SSH into my switch behind the same MTIP region, I am able to access those specific services (SSH, https etc). All subnets at the site:
XXX
XXX
XXX
XXX
XXX
Are affected. In summary, from outside the MTIP region I can only ping devices. From the same MTIP region I can access all ports/protocols as expected.
As you can see, this issues extends far beyond just Missoula Dispatch and is happening across the country. |
