The Radio-over-IP connection between Roseburg Interagency Communication Center (RICC) to both Coos Bay District and Roseburg District radio systems has become unreliable in the extreme since early February 2022. Prior to that time, several months had elapsed without significant outage.
Roseburg and Coos Bay network routers were cut over to Lumen’s network from Verizon without any apparent issue, Coos Bay in September 2021. Springfield router, which hosts an RoIP audio bridge between BLM and USFS RoIP circuits, has not yet been cut over.
1. On 2/7/2022 at approximately 1000 PST, I was notified by dispatchers at RICC that Coos Bay District field personnel did not hear response from RICC to their radio calls. It was quickly determined that even though RICC could clearly hear radio calls, the responses did not activate the Coos Bay radio circuit. RICC personnel thought the problem may have started late afternoon on Friday 2/4 but they weren’t sure.
a. It was also quickly determined that the RoIP network within the Coos Bay office was functional and had been all along, as radio traffic both directions on Coos Bay channels was clearly heard on IP-2002 consoles located within the office proper.
b. The Coos Bay District Office front desk customer contact representatives assumed dispatch check in / check out duties.
c. A helpdesk ticket was opened that day, and a trouble ticket to Lumen was opened the next day.
2. Test calls to RICC from Coos Bay RoIP IP-2002 consoles and IP-224 radio interface were made periodically for the following month. At no time in that period was it possible to hear RICC reply to test calls.
3. 3/8/22: OCIO meeting to discuss plan of action.
a. In the period between 4/7 and 3/8 some effort was made to clean up possible configuration imperfections, with no apparent improvement in functionality. It was determined that the problem was apparently in Springfield, and it was suggested by Lumen engineers that the problem was Verizon’s fault.
b. Lumen people did however acknowledge that there “may have been” changes to router configurations about the time (February) that 2-way RoIP connection was lost. I politely asked, during a subsequent conference call, if it would be possible to change things back and see if the RoIP started to work again. It is my understanding that this request was in fact made known to Lumen.
c. It was disclosed that the identical problem exists in several other DOI RoIP networks; the states MT, NM, and CO were mentioned.
4. On 3/16/22 a request was submitted to Lumen to make cutover of the Springfield router a priority, in hopes that the issue would be automatically fixed by said cutover.
5. On 3/18/22 it was decided to reconfigure the IP-224’s in Coos Bay, Springfield, and Roseburg to Unicast as there did not appear to be any issues when tests were run in that configuration. This change, however, removed all the IP-2002 consoles from the network. It was deemed preferable to have Roseburg and Coos Bay radio reliably connect to RICC than continue with isolated radio systems. (An additional concern was that the radio personnel would not have ability to monitor the network without having a radio in-hand, and this would allow only partial access since most district repeaters cannot be heard from the district offices, but this too was considered less important than having reliable radio contact with RICC).
6. On 4/13/22 @ 1145 PT, following configuration changes to RoIP routers, 2-way communication to RICC from Coos Bay using RoIP Multicast was restored. Roseburg’s 2-way connection to RICC continued to be out, so Roseburg’s router and the Roseburg side of the Springfield IP224 audio bridge remained in Unicast.
7. 4/19/22 @ 0830 PT, RICC informed Coos Bay that they could hear calls from Coos Bay radio users but RICC’s responses were not heard. Test calls were made periodically and at 0925, 2-way communication between RICC and Coos Bay via RoIP Multicast was again found to be functional.
8. 4/22/22 @ 1130 PT, RICC notified the Coos Bay radio tech that their replies to radio calls were not heard. Test calls via radio and using local IP2002 consoles did not hear RICC until 1500. At that time the system appeared to function normally again.
9. 4/27/22 @ 0935 PT, RICC notified Coos Bay that their replies to radio calls were not heard by field users. Test calls at 1030, 1135, 1240, 4/28 @ 0755 confirmed the problem.
Coos Bay front desk assumed dispatching tasks at 0940 approximately.
The 1135 test call and RICC response was heard on Springfield consoles but RICC response was not heard in Coos Bay. It may safely be assumed that identical symptoms existed on the other calls.
The two (4/19, 4/22) outages which mysteriously resolved themselves caused me to think that there was some intrusive testing or reconfiguration taking place without notification. I have no way at present to find out if this was in fact occurring. However, at the time of this submittal (4/28/22 @ 0800) the third outage has not resolved, putting the conditions the same as on 4/7/22 – 80 days!
Note that the outages taking place since 4/13/22 have been progressively longer in duration.
Having front desk personnel performing dispatch functions at the imminent onset of the 2022 fire season is absolutely unacceptable. An interagency dispatch center has information regarding current status and location of all fire agency resources, and has immediate access to the individuals and agencies controlling those resources. In event of a major fire within the areas of interest, the dispatch center can expand staffing as necessary. The front desk people at Coos Bay District Office do not have the same capability. There is nothing at all wrong with the competence and diligence of the front desk people but they simply do not have the training and resources to perform dispatching tasks for incident management and control.
In my opinion the apparent lack of concern, motivation, and sense of urgency on the part of Lumen is a major factor as the obstacle in resolving the problem once and for all. It is hard to believe that nobody in that organization knows what happened on 2/4/2022 nor is capable of fixing the problem that still exists.
Suggested actions to mitigate the problem:
1. Lumen to cut over the Springfield OR router from Verizon to Lumen network ASAP. (Whether or not that specific action corrects the problem cannot be determined until the cutover takes place.)
2. 2-way communication between RICC, Coos Bay, and Roseburg re-established using Multicast. If (1) above does not in and of itself correct the issue, Lumen must not let the issue drop until prompted by DOI, as has apparently been the case until now.
3. Prior notice to affected dispatch centers and/or telecommunication specialists supporting radio communications, regarding any intrusive action to routers carrying RoIP, whether by Lumen or DOI network management. There must be time to advise personnel in the field and to prepare alternate communication resources (if such are available) in case the primary RoIP connections become unusable. |
