ProxyMesh Blog

Reliable Open Proxies

Jacob — Sun, 03 Mar 2013 17:43:23 -0000

ProxyMesh now offers an easy way to make use of open proxies. The typical way of using open proxies involves compiling a long list of IPs that must be updated at least once a day, and using software that can automatically switch between these IPs. These open proxy IPs must also be regularly checked to ensure they're still in operation, and are not inserting malicious code into your requests. ProxyMesh eliminates these requirements. All you have to do is configure your client to access the ProxyMesh open proxy server, and ProxyMesh takes care of the rest. To learn more & signup, go to the ProxyMesh Pricing Page. And for details on how this open proxy server works, read on...

Open Proxy IPs

The ProxyMesh open proxy server maintains a list of known open proxy IPs, which are used by the proxy server to forward your requests. This works just like the US & UK proxies, except instead of going thru low-latency elite proxies, your requests are forwarded thru less reliable & higher latency open proxies. While a large percentage of these proxies are in the US, over half are located in many different countries around the world. These proxies typically do not stay online very long, and are not operated on reliable infrastructure. The tradeoff for this lack of reliability is a huge increase in quantity & variability of IP addresses. The ProxyMesh open proxies list typically countains at least 1000 IPs at a time, with approximately 100 IPs changing every hour.

Realtime Reliability

Because open proxies are not reliable, real-time error checking is required to provide a consistent service. If any request thru an open proxy fails due to a proxy error, that error is recorded, and the request is re-tried up to 3 more times, using a different proxy each time. Any proxy that gets 3 or more errors will immediately be removed from the list.

Periodic Updates

To keep a fresh open proxy list, the proxy list is re-checked every 15 minutes, and any proxies that fail these checks are removed. These checks test the following:

a valid request can be sent thru the open proxy
a valid response is received within 3 seconds
that response has not been corrupted and does not contain malicious code
the open proxy does not have a known abusive IP address

Over 95% of open proxies fail these tests. Therefore, the ProxyMesh open proxy server only keeps the 5% of open proxies that are actually usable.

Retry Strategy

Even with periodic proxy checks & realtime removal of unreliable proxies, you must have a retry strategy when using this proxy server. Open proxies are, by their nature, unreliable & error prone. They could go down at any time, be configured incorrectly, or they may not honor your specific request. For these reasons, you should retry all 40x & 50x response codes at least 3 times, and optionally make use of the custom ProxyMesh headers. The way this could work is:

make a request thru the open proxy server
get a 40x or 50x response error
from the response, extract the IP from the X-ProxyMesh-IP header
retry the request with a X-ProxyMesh-Not-IP header containing the IP

The X-ProxyMesh-Not-IP header can take a comma list of IP addresses, so you can accumulate bad IPs to skip for future requests. If you do this, it is recommended to cache the IPs for a maximum of 1 day, as they will likely be out-of-date or offline after 24hrs.

Elite vs Open Proxies

The ProxyMesh open proxies server provides a very different set of tradeoffs compared to the US & UK proxy servers. With the open proxy server, you get quantity & variability at the expense of speed & reliability. And the US & UK proxies are what's known as elite proxies, in that they remove all identifying information from requests, whereas the open proxies provide no guarantee of anonymity. So if you need speed, reliability, and anonymity, then the US & UK proxies are the best choice. But if you want a lot of IPs, the open proxy server might be an acceptable tradeoff.

Avoid Rate Limits with a Rotating IP Changer Proxy

Jacob — Sat, 05 Jan 2013 16:36:39 -0000

You can avoid rate limits by using a rotating proxy server. For any site or API that uses ip throttling or ip address rate limiting, you can get around those limits if you change your ip address frequently enough. An ip changer proxy, such as ProxyMesh, will rotate ip address for you, making it easy to avoid ip rate limits. A change ip address proxy has the added benefit of also being a hide ip address proxy.

ProxyMesh Bandwidth Price Reductions

Jacob — Sat, 05 Jan 2013 16:36:39 -0000

Bandwidth keeps getting cheaper, and I'm happy to pass those savings on to ProxyMesh customers. The ProxyMesh 50 plan now comes with 50GB included per month, and each additional GB is now only $0.50. This will be a significant savings for all high bandwidth customers, and anyone on a lower plan using 60GB or more each month should consider upgrading. The ProxyMesh 20 plan is now ideal for the 20-60GB/month range, with 20GB included, and $0.80/GB after that. And finally, the ProxyMesh 10 plan now includes 10GB/month.

These changes will apply immediately for all current and future customers. Current customers that wish to upgrade should go to Change Plan or Update Account. And if you have any questions about these changes, please contact us.

How to Change your IP Address with a Proxy Server

Jacob — Sat, 05 Jan 2013 16:36:39 -0000

You can change your ip address on the internet by using a proxy server. A proxy server acts as an intermediary between you and the rest of the internet, so that your IP address changes to be the proxy server IP address. Below are instructions for how to configure your browser or operating system to use a proxy server:

If you are writing or running code that needs to use a HTTP Proxy Server, then checkout these HTTP Client Proxy Examples.

Control the IP Address of the ProxyMesh Proxy

Jacob — Sat, 05 Jan 2013 16:36:39 -0000

The ProxyMesh proxy server provides a few custom headers that you can use to control which IP address to use for each http request. If no headers are passed in, the default method of selecting an IP address is to randomly select from the available IP addresses.

The X-ProxyMesh-IP request header allows you specify exactly which IP address to use. For example, if you pass in the header X-ProxyMesh-IP: XXX.XXX.XXX.XXX then your request will appear to come from the IP address XXX.XXX.XXX.XXX. However, you can only specify one of the current outgoing IP addresses. If you specify any other IP address, you will get a 502: unknown IP address response.

To know which IP addresses are available, use the X-ProxyMesh-IP response header, which is included in every response you receive thru the proxy server. This header looks just like the request header, X-ProxyMesh-IP: XXX.XXX.XXX.XXX, and contains the IP address used to make your request. So you could make a single request thru the proxy server, extract the X-ProxyMesh-IP response header, and then include that as a request header in your next request in order to continue using that same IP address. You can verify that this process is working by comparing each X-ProxyMesh-IP request header to each X-ProxyMesh-IP response header.

The X-ProxyMesh-Prefer-IP request header works very much like the X-ProxyMesh-IP request header, but it will never return a 502 response. Instead, if you specify an unknown IP address, another IP address will be chosen, just as if you didn't include the header.

And finally, the X-ProxyMesh-Not-IP request header allows you to specify a comma list of IP addresses that you do not want to use for that request. For example, if you pass in a header that looks like X-ProxyMesh-Not-IP: XXX.XXX.XXX.XXX,YYY.YYY.YYY.YYY, then neither XXX.XXX.XXX.XXX nor YYY.YYY.YYY.YYY will be used, leaving you with the remaining IP addresses to randomly choose from. It is theoretically possible that you could exclude all available IP addresses with this header, in which case you will get a 503: no available IP addresses response.

These headers are exclusive, meaning that you should not combine them. Any custom header you pass in will only used by ProxyMesh proxy server, and will be removed before your request is passed on to the final site.

Don't Trust Free Anonymous Proxy Servers

Jacob — Sat, 05 Jan 2013 16:36:39 -0000

You may think that a free anonymous proxy sounds great, after all, what's better than free? But be warned - a free anonymous proxy server is no guarantee of privacy protection, and you may be opening yourself up to a malicious operator that will steal your personal information.

No Privacy Policy

A free proxy server that provides a clear privacy policy is extremely rare. Most anonymous proxy servers have no privacy policy at all, and you have no idea what they are doing with your data. While you may be anonymous to everyone else, the operator can track everything you do. On the other hand, commercial private proxy providers are in the business of protecting your privacy, and therefore take it very seriously.

No Customer Service

Free anonymous proxies are notoriously unreliable, and there's rarely anything you can do about it, except find another anonymous proxy. Because the proxy server is free, you are not a customer, just another user. There is no incentive for the operator to provide reliable service, and there's usually no one to complain to when the service is bad, except for all the other unhappy users. As long as you're not paying for the service, there's no reason to expect the proxy server to work well, and it often doesn't.

But for commercial proxy servers like ProxyMesh, it's in our best interest to provide reliable proxy servers and to quickly respond to customer feedback. Otherwise, we lose customers, and a business that loses customers will not be in business for long.

Possible Malicious Intent

There are few legitimate reasons to run an anonymous proxy for free, but there are many nefarious ones. If the operator is malicious, they can watch everything that you do, and save everything that you type into a web page. This is why you should never ever use free anonymous proxies to do financial transactions, or access any site that requires a username and password. Malicious operators can and do extract this information, and will use it for their own illegal purposes, or sell it to other criminals. Basically, you are putting your personal information at risk every time you use a free anonymous proxy from an unknown operator.

Commercial providers, however, want a sustainable business. They don't particularly care about your personal information or what you use their services for, as long you're a good customer that pays on time and doesn't abuse the service. Malicious intent is extremely unlikely, as any commercial proxy provider that was found to be doing something malicious or illegal with their customer's private information would be out of business in a heartbeat.

Reliable Private Proxies

If you want a reliable anonymous proxy, don't waste your time looking for a free proxy server. Instead, find a trustworthy commercial proxy service with a clear privacy policy. Yes it costs money, but the reliability and customer service is well worth it.

Hide your IP Address with a Proxy IP Server

Jacob — Sat, 05 Jan 2013 16:36:39 -0000

Using a proxy IP server allows you to hide your IP address from external sites. By giving you an anonymous IP address, the IP proxy server masks where you're coming from and who you are. Hiding your IP address with an anonymous proxy IP server is essential to protecting your privacy and anonymity online.

Randomly Rotate IP Addresses

A rotating IP proxy, like the ProxyMesh Rotating Anonymous Proxy Server, will change your IP address for each request. Instead of using a single anonymized IP address, a rotating IP changer proxy provides multiple IP addresses to hide behind. Each time you make a request, a random IP address is chosen, making you even more anonymous and harder to track. This makes rotating proxy servers the best way to hide your IP address.

High Anonymous Proxy Servers

Jacob — Sat, 05 Jan 2013 16:36:39 -0000

You don't need hundreds of proxy servers to get high anonymity. Services like TOR and ProxyMesh provide high levels of anonymity by changing IP addresses periodically, instead of using hundreds of proxy servers all at once.

Rotating IP Addresses provide High Anonymity

What makes a system like TOR or ProxyMesh highly anonymous is that the outgoing IP addresses are not the same as the proxy server you connect to, and more importantly, the outgoing IP addresses rotate. TOR changes paths every 10 minutes, which is much faster than ProxyMesh, which rotates IP addresses every 12 hours. However, with ProxyMesh you get 10 IP addresses at once, and a much faster and more reliable service, because it doesn't route you around the world. So if you're concerned with being as anonymous as possible, use TOR. But if reliability and speed are important, you can choose ProxyMesh Rotating Proxy Servers and still maintain a high level of anonymity.

Static Proxy Servers are Less Anonymous

Other proxy services tend to focus on the quanity of proxy servers and IP addresses they can provide. However, these are static IP addresses that do not change. You often get a fixed number of IP addresses, and cannot change IP addresses more than once a month. It's much easier to get blocked using static IP addresses than if you have dynamically changing IP addresses, like you get with ProxyMesh and TOR.

Another reason a proxy service may focus on quantity is because the proxy servers are less reliable, and so you need more of them to make up for it. Some services just provide lists of open proxies, which are notoriously unreliable, slow, and possibly malicious.

Hundreds of Proxy Servers are hard to use

Large quantities of static proxy servers are also more difficult to use. You need to have special software, or go through complicated configuration and programming to access all the proxies at the same time. Compare this to the ProxyMesh HTTP Proxy Server or TOR, which both implement common protocols supported by practially all client software. ProxyMesh even supports IP address authentication, which makes configuration even simpler.

The bottom line is that getting a high level of anonymity does not require hundreds of proxy servers. You can be much more anonymous with a rotating proxy service like ProxyMesh or TOR. And these services can even be cheaper, faster, and more reliable than services that advertise huge quantities of IP addresses.

Paypal Subscriptions

Jacob — Sat, 05 Jan 2013 16:36:39 -0000

ProxyMesh now supports Paypal subscriptions. Once you signup, you'll have the option of paying with a Paypal account or a credit card. Hopefully this will make subscription payments much easier for new customers. And for existing customers, if you'd like to change your subscription options or payment method, please follow the Change Plan or Update Account link from your dashboard.

The Proxy Auto-Configuration URL

Jacob — Sat, 05 Jan 2013 16:36:39 -0000

One of the easiest ways to change your ip address with a proxy server is by using a proxy auto-configuration url in your web browser. A proxy auto-configuration url tells your browser when to use a proxy server, and what proxy server hostname and port to use.

How does a Proxy Autoconfiguration URL Work?

A proxy auto-configuration url instructs your web browser to download a javascript PAC file, normally called proxy.pac, which contains at least one javascript function called FindProxyForURL. Most of the time, this function simply tells your browser to always use a specific proxy server hostname and port for all requests. But the function can also restrict proxy usage based on the request. For example, a proxy server might only support http requests, but not https requests. In that case, the function would examine the request, and if it's for a https url, it would tell the browser to skip the proxy and go directly to the site. However, if the request is for a normal http url, then the proxy would be used. ProxyMesh supports both http requests and https requests through its anonymous proxy server, and so its proxy.pac auto-configuration function will work as both a http proxy server and a https proxy server, but will ignore any non-http requests.

How do I use a Proxy Auto-Config URL with an Authenticated Proxy Server?

Because proxy auto-configuration urls do not support specifying proxy usernames and password (and for good reason, because it would be a security risk), you must use IP address authentication. ProxyMesh supports address IP authentication for its rotating proxy server, which you can configure after you signup for ProxyMesh. Once your IP is registered with the proxy server, then the proxy auto-config will handle the rest, by specifying the proxy server hostname and port for each request.

How do I setup Automatic Proxy Configuration?

First, you need to have a proxy auto-config url to use. ProxyMesh provides all subscribers with this url in the account dashboard, which you can find after you signup for ProxyMesh.

Once you have a url to auto-config the proxy, you must go into your browser's proxy or network settings and look for Automatic proxy configuration, Autoconfiguration url, Automatic proxy configuration url, or PAC File URL, and copy/paste the url. Once you save this url in the settings, your browser will be using the automatic proxy configuration for all future requests.

ProxyMesh Europe Survey

Jacob — Sat, 05 Jan 2013 16:36:39 -0000

ProxyMesh will be expanding to Europe and would like your feedback. If you are interested in a European proxy server, please fill out this form: ProxyMesh Europe Survey. Possible locations for a Europe rotating proxy server are the UK and/or Switzerland, but other locations can be investigated, depending on demand.

Realtime Bandwidth Tracking

Jacob — Sat, 05 Jan 2013 16:36:39 -0000

The ProxyMesh dashboard now supports realtime bandwidth tracking so you can see your current bandwidth usage. This will work whether you're using the US proxy server or the UK proxy server. And if you're using both, the dashboard will show you the combined bandwidth. Another related new feature is that every time your subscription is renewed, your previous bandwidth usage is saved for reference. This can help you estimate bandwidth usage for the upcoming month. And if you have sub-accounts, then past bandwidth usage is saved at the same time, so you can see the past month's usage for each individual sub-account.

ProxyMesh Sub Accounts

Jacob — Sat, 05 Jan 2013 16:36:39 -0000

ProxyMesh now supports sub accounts for all subscribers to the ProxyMesh 20 and 50 plans. Sub accounts are designed for companies and professionals that are using ProxyMesh on behalf of a client, and want to track bandwidth usage in order to pass on the cost. For example, suppose you're a pro at writing web crawlers or screen scrapers, and you've got one or more clients that require the kind of strong anonymity you can get with ProxyMesh. So you signup for the ProxyMesh 20 plan and create a sub account for each client. Now, instead of using your main account, you can use a sub account for each separate client, and see the bandwidth used by each account. Then when the bill comes due, you can charge each client for the ProxyMesh bandwidth used by their sub account. All the bandwidth tracking information you need to do this is available in your ProxyMesh account dashboard.

UK Rotating Proxy Server

Jacob — Sat, 05 Jan 2013 16:36:39 -0000

ProxyMesh now has a rotating proxy server based in the UK! It works just like the original US rotating proxy server, with 10 UK IP addresses rotating twice a day. So if you're based in the UK or Europe and want faster connectivity to ProxyMesh, signup today. And if you're an existing customer and want access, you can now edit your proxy authorizations.

The UK proxy server is currently available for all ProxyMesh plans. If you'd like access to both the US and UK proxy servers, you'll need at least the ProxyMesh 20 plan. For existing customers, you may need to upgrade your plan to enable access. Bandwidth charges are the same for both proxy servers, and if you're using both the UK and US proxy servers, you will be billed for your combined bandwidth usage.

If you are interested in a mainland European proxy server, or any other location, please fill out the ProxyMesh Europe Survey.

Using ProxyMesh with Visual Web Ripper

Jacob — Sat, 05 Jan 2013 16:36:39 -0000

Visual Web Ripper (aka VWR) is a visual tool for data extraction from the web. It allows you to scrape data from targeted web pages without writing a line of code. Though if you do want to write code, you can get full control of VWR using their .NET programming interface.

Using a http proxy server like ProxyMesh with VWR ensures your anonymity and protects your privacy. An anonymous proxy server that changes your IP address will help you avoid rate limits and reduce the chances of being blocked.

To use ProxyMesh with Visual Web Ripper, you must use IP authentication. Once you've subscribed to ProxyMesh, you can go into the account dashboard, and on the right side of the page there is a link to add IP addresses. This link will take you to a form for adding IP addresses (your current IP address is shown to the right of the form). Once you have saved your IP address(es) for authentication, you can configure the proxy settings in Visual Web Ripper or Internet Explorer to use the ProxyMesh proxy server hostname and port.

After IP authentication is setup and proxy settings are configured, you will be able to benefit from ProxyMesh's anonymous IP changer proxy, which rotates IP addresses for you. This will hide your IP address behind a fast anonymous proxy designed specifically for anonymous screen scraping.

Why you should use a Rotating Proxy Server

Jacob — Sat, 05 Jan 2013 16:36:39 -0000

A rotating proxy server provides a lot of great benefits over a normal single proxy server, because it does ip address rotation. This helps you hide your ip address in a similar fashion to TOR, so that each request you make goes through a different, random ip address. A proxy rotator is also more reliable, because it can ignore proxy servers that are not working.

ProxyMesh is an ideal ip rotation service because it handles the proxy rotation and rotating ip addressess for you. It is a proxy server that changes ip addresses. You don't need to do any special configuration or download an app to use the ProxyMesh ip changer proxy, and every 12 hours, new ip address are rotated in and old ip address are rotated out.