Restrict Provision Re-Sync to Off Hours?

[alexnuno]

Hi everyone!

I've been trying to find out if there's a way to get reprovisioning restricted to off-hours. I had a scenario earlier today where we experienced a traffic spike that was noticeable to end users. After digging around, it seems like 25+ desktop phones decided to reprovision at the exact same time, seemingly causing this disruption. On top of this, we have many desktops that daisy chains through desk phones as well, and seemingly were affected. While this only caused a couple of minutes of end-user panic and then subsided, I'd rather avoid it altogether if possible.

After poking around the administrative dashboard, there doesn't seem to be any option for scheduled, reprovisions, or the like. But maybe I missed it somewhere? Or there is an obscure way of doing this? 

Has anyone had success setting up this sort of configuration? Provisioning only during off hours?

Thanks in advance!

[KateG]

Hi @alexnuno nice to see you around the Community. 

Phone provisioning is already scheduled off hours in the pbx's timezone, and usually phones don't all reprovision at the exact same time, time is random in the off-hours window. 

Two plausible explanations for what happened would either be a brief power failure resulting in the phones rebooting or someone hit the resync all button. 

Hope this helps! 

[alexnuno]

Hi Kate!

Thanks for the information and sorry for the delay! I have been keeping an eye on this issue and it seems to have become more prevalent. I had a recent case where the phones are re-provisioned on an entirely different POE switch at around 4-5PM for some users in our warehouse, and this issue has become a much more urgent priority since it disrupts business.

Do you happen to know what determines the "Off-Hours Window"? The timezone on the PBX is accurate... but is there a way for me to manually set any options/ have more in depth information on the PBX/phone side about the cause of these re-provisions(ie, logs that indicate power loss vs provision command from server or anything like that). 

To be honest, I'm at quite a loss. I've been doing diagnostics on the two switches involved in this matter and they are far below POE budget, have uptimes of over 4 months. I'm currently looking into the POE configuration specifics, but it all seems as it should be, and there's no errors logged on our them that indicate POE overload/failures, and one user that experienced the same issue even had a separate POE desktop switch that they were using, not one of our core ones.

For now, I have deployed a new POE rack switch for a couple of users to test out, and provided device level POE injectors/dedicated power supplies for others in case that helps alleviate the issue while I further isolate the cause. 

Any and all thoughts and suggestions are appreciated.

[KateG]

Hi @alexnuno nice to hear from you, though sorry to hear this is still occuring. 

Off-hours are 11 pm to 5am in pbx's timezone. The next best bet would be to work with support  so that we could look at logs on our side. I've gone ahead and created a case for you and someone from our support team will be reaching out. 

[alexnuno · Accepted solution]

Hi Kate!

Thanks so much for this! I worked with support yesterday and think we found the solution. Apparently it's surprising that our phones were working at all given how they were deployed. There were a couple of core GTC Firewall rules that needed to be set, as per the deployment requirements. Either these were never set in the first place, I inadvertently affected rules at some point, or a recent firewall update messed with our settings.  Apparently this is a semi-common issue that at least the tech I was assigned has seen around, so I am praying that this is indeed the fix. 

1. SIP ALG needed to be disabled.
   1. https://support.goto.com/connect/help/what-are-the-recommended-sip-applications-settings-gotoconnect...
2. UDP Keep Alive needed to be set to 300 seconds. 
   1. https://support.goto.com/connect/help/what-are-the-recommended-nat-keep-alive-settings
3. Ports 5060-5061 needed to be allowed through the firewall. 
   1. https://support.goto.com/connect/help/what-are-the-required-router-settings

The changes have been in place for 24 hours, with no mass provision yet. 

Fingers crossed. 

Thanks again for all of your help Kate! Best to you!

[KateG]

Hi @alexnuno great to hear from you, and I'm happy to see you were able to find a resolution. 

Thanks for taking the time to share the solution here on the Community, so any others who run into this problem can benefit from your action. Have a great day!