I'm seeing a consistent pattern of increased latency and packet loss every evening between ~6pm and ~12am. I do pings every minute and record the latency and packet loss.
The graphs are from the last 24 hours and you can see this spike last night. If it was just last night I wouldn't be worried but looking at the last 30 days:
You can see that up and down that is happening every single night. I've seen this pattern since Novemember 8th. I don't believe this is on my end. This runs on a wired connection from my raspberry pi to my router and out to the internet. I've never seen this pattern before.
Do you see anything on your end? Is there something that is causing this type of behavior?
Thanks,
~Jeff
Thank you for following up. The last two days I haven't seen it happen. It seemed to stop on the 26th of November. It has maintained (or slightly improved) the ping times. I still see packet loss from time to time, which is still concerning. This is the last 24 hours:
How this manifests to me is that things will randomly not load (or seem to be stuck) on a web page but if I refresh then it will load just fine. If something is loading it loads really quickly.
I ran mtr to google.com while I was having the issue and while I wasn't, and nothing really stood out to me as being different between the two time periods. The only thing I thought was interesting was that ip 67.199.171.241 which is before cognet seems to have consitent intermittent packet loss. My first 2 ips are always 38.78.248.1 and 67.199.171.241.
But with mtr it is hard to know what the behavior should be at each hop. Running `traceroute 8.8.8.8` I see the same on and off behavior with 67.199.171.241, where it didn't get a response the first time or the thrid but did the second.
❯ traceroute 8.8.8.8 traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets 1 firewalla.lan (172.16.1.1) 1.362 ms 0.539 ms 0.456 ms 2 38.78.248.1 (38.78.248.1) 1.931 ms 1.625 ms 1.448 ms 3 * 67.199.171.241 (67.199.171.241) 9.974 ms * 4 * * * 5 be3110.ccr22.sfo01.atlas.cogentco.com (154.54.44.141) 19.477 ms 19.084 ms 18.683 msIf there is anything I should be doing specifically on my side to look at something, just let me know.
Thanks,
~Jeff
Test results from last night:
I did record an increase of latency around 6 PM last night but the increase was clearly being caused at Google’s router. My system and Cogent, our provider, were flawless. Pings increased from about 40 ms to google dns 8.8.8.8 and 8.8.4.4 to a high average of 57 ms. The ping times were still well with acceptable ranges.
This is what a traceroute showed at 4 PM and at 8 PM but I have the data for every 30 seconds from 2 PM until the next morning:
Sun 26 Nov 2023 04:00:47 PM MST
traceroute to 8.8.4.4 (8.8.4.4), 30 hops max, 60 byte packets
1 _gateway (128.254.148.1) 1.445 ms 1.377 ms 1.350 ms
2 67.199.171.241 (67.199.171.241) 1.119 ms
3 *
4 be3110.ccr22.sfo01.atlas.cogentco.com (154.54.44.141) 18.154 ms 18.131 ms 18.109 ms
5 be3179.ccr22.sjc01.atlas.cogentco.com (154.54.43.150) 19.203 ms 19.182 ms 19.159 ms
6 be3177.ccr42.lax01.atlas.cogentco.com (154.54.40.145) 31.251 ms 31.306 ms 31.099 ms
7 be3359.ccr41.lax05.atlas.cogentco.com (154.54.3.70) 31.055 ms 31.216 ms 31.150 ms
8 tata.lax05.atlas.cogentco.com (154.54.11.194) 30.935 ms 30.831 ms 30.759 ms
9 *
10 142.250.164.40 (142.250.164.40) 41.429 ms 42.396 ms 41.336 ms
11 *
12 dns.google (8.8.4.4) 40.548 ms 40.520 ms 40.850 ms
Sun 26 Nov 2023 08:00:05 PM MST
traceroute to 8.8.4.4 (8.8.4.4), 30 hops max, 60 byte packets
1 _gateway (128.254.148.1) 1.408 ms 1.334 ms 1.489 ms
2 67.199.171.241 (67.199.171.241) 1.251 ms
3 *
4 be3110.ccr22.sfo01.atlas.cogentco.com (154.54.44.141) 18.119 ms 18.166 ms 18.070 ms
5 be3179.ccr22.sjc01.atlas.cogentco.com (154.54.43.150) 19.353 ms 19.332 ms 19.309 ms
6 be3177.ccr42.lax01.atlas.cogentco.com (154.54.40.145) 30.989 ms 31.266 ms 31.216 ms
7 be3359.ccr41.lax05.atlas.cogentco.com (154.54.3.70) 31.428 ms 31.398 ms 31.384 ms
8 tata.lax05.atlas.cogentco.com (154.54.11.194) 34.484 ms 30.943 ms 30.875 ms
9 *
10 142.250.164.40 (142.250.164.40) 57.770 ms 58.791 ms 57.842 ms
11 *
12 dns.google (8.8.4.4) 56.354 ms 57.645 ms 57.161 ms
Notice the times were about the same on my router and through Cogent but increased when it hit 142.250.164.40 which according to ARIN is a Google address range.
There was the occasional packet loss but didn’t appear to be significant. Loss is normal and was about what I would expect and was not likely caused in our network but I will test that further tonight.
It doesn’t look like anything should have been having a significant impact on performance and our network was flawless. Are you seeing an actual performance decrease?
I will do further testing and post the results. If anyone is interested in getting all the raw data just email me and I will send it. johnb@salemcity.org (assuming you are a Salem Fiber customer)
Thanks,
John Bowcut
Director
Salem Fiber
385-437-4411
I will do some testing but nothing on our end should cause the issue and I haven't been seeing the same performance issues. We would have to be saturating our bandwidth and we aren't even close.
I will post again after my testing.
John Bowcut
Director
Salem Fiber