Internet Limiting / Restrictions - Does Your ISP Tell You the Truth?

What will this Instructable be about?
Firstly about some technical stuff in regards to your connection,
secondly about how to check your connection in ways your ISP does not want you to know.
But most importantly about how to figure out if your ISP is limiting your connection or blocking certain services.

Do I have to read it?
No, not really!
But if you think your service is not what you pay for you might find some useful information here.

First some terms I will use throughout:
ISP - Internet Service Provider, the guys you pay to get the internet running.
Protocol - there are many but you only need to know that they regulate certain internet traffic for now.
Provider - same as ISP.
Modem / Router - the box you connect to so you can access the internet, can be wired or wireless.
Port - Everything on the internet uses a so port to transfer the data, HTTP uses port 80 for example.

Read on for a little game of true or false....

Step 1: True or False? a Little Game :)

Noone really reads all the fineprint when it comes to contracts with an ISP (Internet service provider).
All we care about is the price, the speed and in some cases what additional extras, like internet TV, Voice over IP, to receive.
But what if something goes wrong or does not work as expected?
Let's play to find out:

T/F) Your ISP will always give your the best service possible.
Both true and false, your ISP is mainly interested in a good quality service for everyone.
In times of congestion (when the ISP sruglles with demand) your ISP will slow down certain all internet services as otherwise some customers might be unable to use their internet.

T/F) Internet is Internet.
False. For you the internet might be playing online games, watching Youtube or downloading the latest trailers.
For your ISP the only services that really count are Email and HTTP, or web browsing.
Downloads using HTTP are what is considered the standard, file hosters or P2P services, including the Usenet are "luxury".

T/F) If you have a problem with your speed your ISP will help to fix it.
Only true if it affects your normal surfing or Email.
In all other cases they will refer to their test and online speed tests to confirm you have no problem.

T/F) Your ISP will tell you on request if they are limiting your connection in any way.
False! As above only HTTP and Email are of their concern and you will see in your contract that during times of congestion you might experience slower speeds.
Some contracts still use so called "off peak" and "on peak" times and only offer full speed at off peak hours.
So if you complain to your ISP that your Torrents come to a crawl he won't even listen, let alone tell you they know what you do and limit your speed.

T/F) Your ISP will protect your identity.
False! Thanks to US regulation and demands by security agencies (again mostly US based) your ISP will give out all details if asked by the authorities.

T/F) Ok, fair enough if I do something criminal but else?
Still false! These days all your traffic is analysed and checked one way or the other.
Sending some fun Emails to firends containing certain key words or phrases will flag the mail.
Although you won't know anything about it, your ISP is required to hand out all information even if all is just joking with your firends.

T/F) Your ISP is checking what you download, where you get it and what websites you visit.
False for most countries but true for the US, China, Korea, Australia and some other countries.
The EU is considering restrictions too.
Due to the anti terrorist crap imposed on us the US insists on their right to know what everyone on the planet is doing online.
Your ISP is supporting those effords by providing easy access to the traffic and if required the user.
Basically it means that your ISP is preparing all data for security agencies in a way that is free from obfuscations.

T/F) Your ISP always provides the best hardware for your type of connection when you sign up.
False. Most ISP's use outdated modems / routers as they are cheap to get.
In many cases the installed firmware will be modified to the needs of the ISP as well resulting in so called "crippled hardware".

T/F) Your ISP will trust you when you tell them you found a certain problem with your service.
False. You ISP will use a call center and a guy or girl reading a spread sheet to check your service.
Usually these "sessions" are designed for the DUP - Dumbest User Possible.
So even you confirmed the problem by connecting your mobile to the computer to act as a modem for the 3G or 4G network you will star from scratch with your ISP.

T/F) You can not get someone with decent technical knowledge to help you.
False! :) Most ISP's offer a customer service called "Level 1 support" or refer to their technicians upon request.
For severe or ongoing problems it often helps to do it blunt and as for a "ticket" for higher level of support.
Although not shown anywhere on the websites they will give you a phone number and often also a ticket number for your case.
There you have people that can access your connection for tests unlike the call centers.

T/F) I am a heavy downloader but my connection is limited, a provider change will help?
False. Most ISP's will provide a totally unlimited service for a month or even three months before applying any limitations.
Although in heavy cases these restrictions can come sooner.

T/F) ALL ISP's can provide the same speed and quality of service.
Ok, you guessed it, flase again...
The network is owned by one company, usually they also provide all phone and internet services.
All other ISP's are so called retailers.
The buy or rent parts of the network in some cases but mostly just just do this for services.
This obviously means the company owning the network in your area will make sure their customers get the best service possible.
Retailers can be limited in speeds or blocked for certain services in times of demand - apart from the problem that they have to make sure to stay within the boundaries of their contract with the network provider.

I don't know if you liked this little game but it is time to move on now.

Step 2: Checking and Ruling Out Other Problems First

Ok, I get it! You connection is not the best and you are desperate to know how to address the problem, so let's get going here.

If your connection is somehow flawed it always pays off to keep an eye on it for a while.
Here is a little checklist:
Does the problem happen all the time or only during certain hours of the day?
Is it a general problem affecting all internet services or only things like Youtube, P2P or video on demand?
Did you check if other phone outlets for your modem / router work better?
Did anything change on your system before your noticed the problem? Like Windows updates, new programs or hardware installed....
Was there heavy rain or snow?

Starting from the top...
Speed problems during certain day time hours or on weekend are usually caused by congestion.
Congestion means too many people are using the internet in your area at this time.
Your ISP will limit certain services or your speed to ensure everyone can still use the internet with "sufficient" speed.

If only certain services are affected, mainly P2P traffic and streaming services but normal speed tests result in good speeds your ISP is limiting these services.
It can be a thing of traffic in general, like you being on a 512k line trying to to watch high definition video on Youtube.
But also the fact that your content is not considered normal traffic, like P2P services.
In some cases this happens at certain times only in some case a limit disapears if you restirct yourself not use the service in question for a day or two.

Believe it or not, problems with the wiring on your phone outlets are more common than you think.
Same for failing DSL filters.
So if your problem happens at all times and for all services check if the outlet is faulty by testing a different one in the house and if possible test a different line filter as well.

Updates or a change of hardware can affect your connection as well.
It always pays off to double check if new hardware or any sofware updates were made, this is true for the firmware of your router / modem as well.
A new driver or service running in the background can be overlooked ;)

Last but not least the weather.
If you are not on cable the weather can be one of the biggest culprit in ruining your internet experience.
A satellite connection suffers in heavy rain or snow, not to mention thunderstorms.
In some cases spider can enter the LNB and block the signal.
A local area WiFi (common for remote communites) will suffer under heavy usage, bad weather and radio interference if there are not enough free channels.
But even you phone line is not fail safe.
Problem here is the so called "last mile" - everything from the local exchange to your phone outlet.
If a connetion point gets flooded by water it can not only corrode the connecing cables but also transmit electricity between them, this causes signal loss and interference.
If your problems always happen during or shortly after very bad weather and disappear once the weather is fine request a physical line check from your provider.
In most cases they will tell you that you have to pay for the work unless there is a fault found - so don't use it lightly.

Step 3: How to Actually Test for Limitations...

As I said before your ISP is only interested in the normal traffic but not in streaming or P2P (there are more I know!).

What you do not know is that basically all websites hosting speed tests are prioritised by internet providers.
That means you can experience (legal) very slow download speeds but the speedtest show you are able to reach at least 90% of the speed you signed up for.
For a real world test you should try downloads from different sources around the globe using fast and reliable servers.
Linux distributions are a good way as often they are available as HTTP and P2P (torrent) downloads.

But of course nothing of this will help you to actually know what is happening to your connection.
A good indicator you can install for Windows is a little gadget to show your connection speed as a graph.
Traffic limiting or throttling usually happens in short intervals rather than a flat capped speed.
There are several different one available, I use Network meter

Check the below pic:

What you can see in the the above picture is a classic case of P2P traffic limiting.
You notice the spikes going up and always at the same interval they drop down?
It is called "packet dropping" and the easiest way to slow you down to a crawl.
If this happens while you do normal HTTP download your download will be most likely corrupted and unusable.
You might also notice the download will stop before it finnishes.
Packet dropping can be done on you entire service but in 99% of cases it is limited to ports and protocols not used for standard internet traffic.
UDP ports and those causing high numbers of simultanous connections (P2P) are prime examples.
In the above pic you also see the speed never makes it up high, this is because here we deal with both timed and speed capped traffic shaping using packet dropping.
Packet dropping means your provider will simple stop sending the data information and loose it, your computer has to restore the lost data by requesting it again and again.
The result is the drop in speed to almost zero as in this time the data is checked and the information about the missing bits collected.
Then is speeds up again to continue only be be blocked again....

Here you see the same but using a HTTP download:

You see the same distance between the spikes but they no longer drop to zero and the overall speed is much higher.
I could have gone higher but got a slow download for this one.
But you still the speed shaping taking place.
The big difference is that a packet drop on a HTTP download is far less damaging than on a P2P download or video stream.
In the above case you can tell that traffic shaping / limiting is happening as it is the same pattern for the spikes.
But of course you have to confirm this several times with different download sources.
A normal download goes up and down in speed as well but usually would not result in this nice mountain line in the speed monitor.
It would be more like a wava going up and down slowly withuot any sharp edges or spikes.

Amore detailed test than any normal speed test you can find is provided by M-Labs.
This the the running NDT (Network Diagnostic Test):

And the result:

Compare this with a standard speedtest on the net (Speedtest.net):

As you can clearly see my speedtest using M-Labs shows the traffic shaping happening while the standard test on the web maxes my connection to the brim.
Notice that the upload speeds are almost identical? ;)
Your ISP will always refer you to one of the online speed tests, but never to one like M-Labs that would show your real speed based on a mix of different protocols.
At times the M-Lab test will show you slow speeds although your downloads are looking fine, it simply means it is detecting the shaping but right now it is not affecting you in your speed (ISP might have enough bandwidth to spare at the time of the test).

Here you see the network monitor at off peak hours:

I just started a download and the speeds are going up but the most remarking difference is the the lack in evenly spaced spikes interrupting the service.

The M-Lab test still indicated the same as above so please combine both checks in your own tests.
Another nice test provided by M-Labs is "Glasnost".
If you internet connection is all good it will give you a result similar to this one:

This test was done in off peak hours and for the transfers Emule uses, you can select other tests by following the above link.
The above image shows the test was a success, if there is a problem you could see it in the results with more detailed information.
If you need an image for such a case let me know in the comments and I will see if I can get a test done while shaping is happening.

Last but not least for real world testing is the Netalyzr:

As you can see it also available for Android.

You don't have to click or fill out anything while the test is running if don't like the idea of doing so.
In return you will get quite detailed results showing you what is happening with your connection and if any traffic shaping is happening or ports are blocked.
Please be advised that the test requires Java and proper settings for your firewall / anti virus software so it is not blocked.
Here is a results screen from the tool:

Apart from the UDP test the rest is fine.
I say fine as basically all red items are blocked by my firewalls and I did not add an exclusion for the test.
All HTTP and DNS sections should show "OK", if not you have a serious problem.
If you need to check the above red listed items your have to add exceptions to your firwall configurations.
Although this is only required if you really encounter any problems on these services.
In some cases the Android app can be helpful to rule out restrictions by your computer.
Especially the "Network buffer Measurements" test in the Network Access Link Property section will indicate problems with traffic shaping.
If it happens (and the test is executed) you will see something like 1800ms buffering for the Uplink.
This is the actual traffic shaping happening.
So if in doubt use the Android version of the test and check through your WFi network.

Step 4: Ok, I Am Limited - What Can I Do ??

In reality not much.

For P2P downloads it can help to limit your sources to under 50 and the open onnections to the same value or lower.
If you client allows also limit the max connection for the entire session.

Streaming can be improved by checking a different DNS provider like Google 8.8.8.8 and 8.8.4.4 or OpenDNS 208.67.222.222 and 208.67.220.220 - this is helpful if also your webpages take a long time to appear.
By that I mean you see in the status bar: "Waiting for..." while not much is happening in your browser window.
If you can using a lower bandwidth or resolution (480p instead of full HD 1080p) might give you a fluent video and audio at the cost of image quality.

If you normal and legal HTTP downloads are affected by limitations or traffic shaping it can result in dropped downloads, corrupted downloads and generally a big waste in your internet traffic.
Download tools like Free Download Manager or the Browser Extension "Down them All!" (just to name some) can help you there.
Especially if a checksum of the file is provided.
You can add the checksum to the download manager and if it does not match it will correct or re-download the file(s).
This is especially helpful if the host allows to continue broken downloads as the download manager can seek out the broken parts to finnish a complete download.

For traffic shaping due to congestion, like when it only happens at certains hours of the day you won't find a solution other than changing the provider - which might not even help in the long run.

Of course some will now point to the Usenet, VPN servers or even services that re-direct and encrypt your traffic, not to mention the Tor network.
I did not mention those "services" as their main purpose is to hide your identity and therefor have nothing to do with traffic limitations imposed on you by your provider.
If you are in a country that limits the content of the WWW by censorship and you fear prosecution for accessing it they are good.
You ISP will still know that you are up to no good and if in doubt simply limit everything and every port that is not used for normal Email and internet browsing.
And I am not going into the details of personal safety for illegal downloads in why all those services won't really help you once target by security services.

You might think about complaining to your provider but I can tell you once you mention anything that can be used for illegal purposes as well they won't care at all.
If you are streaming freak it is often hard enough to make them open the ports you need for your internet TV and to make sure you get enough bandwidth for good viewing quality.
Even if your are on an "unlimited" internet plan with high speeds and basically do no downloads of files, games or movies.
If you really want to complain be prepared and only complain about services that are entirely legal.
The excuse that you can not download your Linux distribution through torrents will fail as they see the Gigabytes in traffic you collect and from what sources they come ;)
But if you constantly encounter unusable HTTP downloads, like damaged archives, EXE files that won't open and so on you will have a good claim.

I hope you find some of the informations I provided usefull for evaluating your internet problems.