Exploring Network Change Error
The user experience of my web app was getting affected as they are facing ERR_NETWORK_CHANGED error.
And this is the story of the investigation I did, conclusions were data-driven to remove any subjectivity.
Now, there can be many reasons which may be resulting in this error such as browser, Internet speed & ping, ISP, AWS, or Docker networks.
The first question that came to me was, is there a bottleneck at the server? I analyzed the logs of the webserver and AWS network, there were no anomalies in them.
To reduce the variables in the equation, I removed the apache from the server. Apache2 was handling requests coming to the server and routing them to caddy.
Next in line was Docker. After little googling, I found disabling ipv6, restarting containers, and networks are the things that people are exploring and claiming to be a source of the problem.
I checked and docker containers weren’t restarted at that time. Pruning networks didn’t help and the same with disabling ipv6. So, Docker was out of the equation for now.
Upon searching more, I got another lead i.e. chromium, and there were a lot of claims in this direction with explanations. I tried my web app on firefox and it looks like the issue is actually with chrome.
But without double-checking it with data, I can’t be objective about it. I opened my analytics tool and mapped the network error with respect to their browser.
And all the browsers are getting this error. So, it directly eliminates this browser theory.
Location was also eliminated very quickly, as the locations of these errors were from pan-India.
I reached out to the users who were facing it very frequently and recently, to gather a few details. like internet speed, ping time, and their ISP.
Internet speed was a minimum of 20 Mbps with 60ms as the highest ping, I concluded that these should also not be the source of the issue.
For ISP data I tried to correlate these error events with ISP users' internet reporting of issues with their service. There were few citings but that was not concrete as distribution of ISP was very high and expecting, all of them to get affected at the same time looks more like a coincidence.
The problem still exists and I am still finding the root cause. I wanted to share my knowledge in the open so that someone who is also looking into this problem can find it.
Please comment or reach out to me, if you have any suggestions.
