Connection Monitoring Software

My last post "Connection Quality" went into technical detail about how a slow Internet connection can be troubleshot to determine the cause of the slowness.

So here is my idea on a software program that will work with any application that uses the Internet for connectivity such as streaming multimedia, remote connectivity such as Citrix, VMware View, RDP, and online gaming to name a few.

Lets start with an example of a Citrix session using the ICA protocol. A student is in a coffee shop and connects to the schools Citrix farm where they have access to an online portal with different apps for their classes. The student has very poor performance and calls the help desk. After some troubleshooting over the phone the help desk determines that it is the Internet connection at the coffee shop.

Now lets introduce the concept of software the not only monitors the quality of the connection but will also pinpoint where the issue is.

A small piece of executable would run as a service or in the background all the time. This would sense when a network connection is open that the software has the ability to monitor. It would then present a small graph or meter overlay on the screen somewhere not in the way. This would only show when the connection first starts or when the quality is bad or changes. These display settings would all be configurable including the thresholds to what is a "bad" or "good" connection.

Because the Citrix connection is connecting to a server at the schools data center we need to monitor the entire path. This means pings, traces, quality of service, throughput on the protocols being used, dropped packets etc all along the route. The software may identify that the issue is with the router or Internet connection itself at the coffee shop. The software is smart enough to not only test the speed and latency from inside the coffee shop but also ask an external server to test the path from outside the firewall. The results are then compared and it verifies that in this instance it is indeed a slow connection at the coffee shop.

The user is then notified that they should use a different Internet connection otherwise they may have a very poor experience with their Citrix connection.

Now lets say the issue was downstream with another router. The software would identify that and alert the user that there is an issue along the path and they may notice a slow connection. It will suggest trying another Internet connection. If the route is nearby possibly allowing them through a proxy (likely a premium service that the school might provide or the user can pay for) that will take a different path and reroute their traffic around that problem router.

All of this information is helping the user, reducing help desk calls and more importantly is reporting anonymously back to the central servers what routers are causing issues. ISPs all over the world can subscribe for free to this database for alerts. So when a router they are responsible for is an issue they will be notified. Thus improving the experience for everyone around the world.

It is even feasible that the central servers when they see a large amount of issues on particular routes can automate signals to subscribing routers to "poison" a route making routers go around that route instead of continuing to be a problem.

A website that would be accessible to the general public would graphically represent issues around the world as well as general statistics to what sites are busy, what games people are playing the most, etc, etc.

The primary revenue stream for this product would be that the software itself is paid for by the end connection point owners. In this scenario the school. The end users would benefit from being notified of a bad connection and the connection point owner would have reduced help desk calls to troubleshoot bad connectivity. Additional revenue streams such as a proxy to go around troubled routers and provide caching services could be paid for by the end user. In this case the student. If the school wanted to provide this service they could provide it only when connecting to their services.

I would suggest a free version that offers basic monitoring of the connection quality but does not get down to the protocol level and will only monitor out to a few hops. This will keep check on the quality of the local Internet connection. If they connect to a resource that pays for the service it will automatically engage the full feature set of quality monitoring and proxy services if that is also paid for.

Connection Quality

The world today is so connected that a computer is almost useless without an Internet Connection. But many people do not have a good connection to the Internet. More specifically the route they have from their computer is to the backbones of the Internet is slow. But what is slow? A slow or poor Internet connection is based on many factors but ultimately it is the perception to the end user of having to wait. Here are some common issues that cause "slowness":

Latency: How long it takes for a single packet to make a round trip in milliseconds. Less is better, anywhere from less than 1ms for a local network to 200ms for an Internet connection. Typically you want this to be under 60ms for a "good" connection.

Jitter: The amount of delta from one ping to the next, generally this is an average. Less is better, under 20ms is good for most connections.

Packet Loss: How many packets are lost in transmission. Less is better, should be well under 1%.

Hops: How many routers a packet must travel through. Less is better. There is more chance of packet loss with more hops.

As you can see there are many factors involved in that little TCP packet that goes from your computer to the server for data and back. TCP is a reliable protocol whereas UDP (normally for streaming video and music) is an unreliable protocol. TCP will re-transmit data when it does not receive it. There is also a handshake from end to end to confirm that the data was received. Along with this is check-sums to prevent corrupt data or incomplete data being received.

All this extra payload requires some overhead. Most do not realize that there is upload bandwidth used when downloading and vise versa. So when your ISP says you have a 10 Mbps download and a 512k upload, it is possible that your download speed is actually hindered by the upload limitation. For those that want to understand the technicality of this here goes...

A typical packet size is 1500 bytes (jumbo packets are larger and we are not going to talk about that here). A header is typically 40-60 bytes. The header has information on where the packet come from, where it is going, and what is in the packet. If the header is 40 bytes then that leaves 1460 bytes for data. This means a minimum of 2.7% overhead. Now if you are not using all the space in the packet, lets say only 100 bytes of data is sent then the overhead would be 40%. If you are doing something like playing an online game that is sending lots of small pieces of data you could have a high overhead and now your Internet connection is saturated with lots of small packets talking back and forth.

Programmers are smart and are pretty efficient with the transmission of packets. So if there is a request of data of say your position in an online game, instead of sending 10 small packets the software might combine the data request into 1 larger packet. Thus less overhead and if the connection is bad it is faster to retry sending 1 packet vs 10.

SOOOOooooo what can you do to improve your connection speed. There are lots of tricks out there including changing your MTU (maximum transmission units or packet size), installing software that promises to make your connection faster, or simply get a better ISP. First of all the MTU size is not likely going to do much as most network equipment will automatically pick the best MTU. Software installed on your computer is generally a good way to get malware or junk software. The best solution is to get the best ISP you can.

Money is not everything, so paying for the fastest connection might be a waste of that money. When you pay for a connection and there are different tiers of speed all the ISP is doing is "capping" the speed based on what you pay. For example if a cable company will provide 10 Mbps down and 2 Mbps up (I will refer to as 10/2 going forward, or down/up), 25/5, and 50/10. These speeds are controlled by software. So when your connection approaches these set limits it will slow you down. This is also known as packet shaping. Also some connection have a "burst" where the first say 10 MB (that's mega bytes) of data will download twice as fast and then it will slow down to what you pay. This makes customers happier and makes speed test results much nicer too.

The problem is the connection quality is the same, you just have a speed limit. Think of it as driving around a fast car but your not allowed over 100 mph or 80 mph. Do you really need to go that fast? If you check your email and surf the web then you need a basic speed connection. If you stream movies all day and multiple at a time then you need the faster speed. For gaming it really does not matter as the bandwidth is generally not that high. Latency is a more important factor.

How can you check quality of your connection? Here are a few sites that I use that will help.

http://www.pingtest.net/ (requires Flash and Java)

http://www.speedtest.net/ (requires Flash)

http://speedof.me/ (requires an HTML 5 browser like Chrome, Firefox, or IE 10+)

But these test are not showing the whole picture. They are testing your connection from your computer to their servers. What if the problem is not your connection. What if the issue is a router between you and your destination? A few quick troubleshooting test will help with this. Here is what I do to check my connection.

1. Go to several large websites to see if they all load slow. Like Google, Yahoo, MSN, Ebay, etc. If they all load slow then it is likely that the issue is with your computer or your connection.

2. Try another computer on the same network. Again if the same issue occurs then it is likely your connection.

3. Start a trace. In Windows open a command prompt (go to the run dialog and type CMD then press enter). Use the command tracert and then an address like this (without quotes) "tracert google.com". You will see a list of addresses and some will have names next to them. These are the "hops" or routers that your connection is sending packets through to get to Google.com. If you see an * meaning no reply don't worry as some routers will not reply, that is normal. Look down this list, if you see very high times in ms like well over 100 then that is likely where the slow down is. If those high numbers are at the beginning of the list the issue is close to you. If the numbers are near the bottom the issue is more toward Google's end.

4. Path Ping is another command that is good for troubleshooting. It is like a Trace Route but will do more analysis over the path with lots of pings. This takes longer to run but can provide more detailed information on where there is an issue.

5. Reset your network equipment as that generally never hurts. Do some of these test again to see if the problem is still there. If it is then you might have to contact your ISP. If the issue is further downstream then sorry there is nothing you can do about it. Just wait and try again later.

Each time you make a connection the path may change. Routers are constantly talking to each other to find the best route. So if one router goes down or a link is really busy they may all send out signals to find a better path. This could change the route and will "fix" that problem you had. Go ahead and try it. Do the same trace every day and at different times. You may find that the path changes a bit all the time. As ISPs add more bandwidth and more routers a better path may show up and the routers you talk to may send you on your way.

In conclusion the Internet is truly an amazing beast with massive redundancy everywhere. To take down the entire Internet is just about impossible and has been tried. Several attempts have been made to do this by targeting DNS servers around the world. This has caused slowness but never actually totally made the Internet inaccessible. DNS is what resolves a name like google.com to an address like 173.194.33.165. Think of directions to a house. If you know the persons name but have no address you will never find them.

So the next time you have slow Internet take a few minutes to understand what the issue is. Almost always it is going to be at end or with your equipment. Before you call tech support or freak out. Reboot your router, modem, and computer, basically reboot everything.