To the Swift: Proxies on Your Relay Team, Part One
Proxy services offer many advantages: hiding your IP address, providing high anonymity, screen scraping, avoiding rate limits and blocks. In all of these advantages, proxy speed is a substantial factor. You might think of the proxy as a key runner on your relay team.
This two-part article focuses on:
- Measuring and maximizing the speed of the proxy as a component of the relay.
- Things you can do to speed up processing of a proxy request.
For discussion of other components, please follow the links in this article.
So, are you auditioning proxies for your network? You’ll want to know how fast they can run.
Proxy speed greatly affects the pace of the round trip for an Internet data request. Commercial proxy services advertise the processing speed of their servers, commonly measured in seconds or milliseconds. Actually, the round trip involves many other components in addition to the proxy. It starts with the internet user, passes to the proxy server, then to the target remote site, back to the proxy server, and finally to the originating user.
Measuring the speed of this journey is a complex process. But whether you’re shopping proxy services or assessing one you already use, you can find tools to understand and measure the factors in proxy speed. You’ll observe varying transmission rates among proxies you may connect to. And you can find ways to increase proxy speed, or keep from slowing it down.
One site offering useful background on proxy speed tests is testmy.net, which also provides tools for testing the speed of uploads, downloads, and other factors. https://testmy.net/country/a1/max, can show you the speed performance of anonymous proxies in different countries.
Another informative site is the Thousand Eyes blog. You may want to take particular note of their article Measuring Performance with HTTP Proxies.
And, if you use the FoxyProxy add-on with Firefox, check out the knowledge base article Testing Service Speed. Note that in a test this service will choose a remote site close to the proxy server you’re using. For example, if your proxy is in The Netherlands, the testing service will target a remote site in that country.
Lots of Do’s – and a Few Don’ts
Can a proxy server speed up your internet transactions? Opinions vary. Some say a proxy will actually slow your connection down, because proxying adds at least one more server to the journey your request will make – a tradeoff for anonymity, privacy, and avoidance of bans, among the other proxy advantages. Others say a proxy only makes your Internet seem slower – or faster – when it connects to a remote site for you.
But we’ve just seen that objective speed tests can give you real answers. And there are many things you can do to measurably speed proxy transmissions or keep from slowing or stopping them. We’ll cover them in Part Two of this article. But first, a word about things you can’t do.
Extra IPs on Server
Proxy servers usually have a limited number of IPs, although that limit could be in the tens of thousands or even more. And they also may limit concurrent connections per IP address. Some users running large projects may require more than that at a given time.
One option is to slow your crawl rate on your current proxy. Another is to use multiple proxies so you’ll have more IPs running simultaneously. It’s good practice to distribute your requests across multiple proxies.
Persistent connections – those that stay open for more data exchanges after a first HTTP request and response – offer certain advantages, including reduced usage of CPU and memory. However, proxy services are typically not designed for persistent connections (that’s what a VPN is good for). A proxy connection normally closes after transmission of just one request and one response. The proxy service may detect and disable a keep-alive header included in a request.
- Try a different proxy, with a nonpersistent connection, if you think the one you’re using may be unreliable.
- Use a timeout header if you need longer for a request.
- Have a good request retry strategy.
Enough about what you can’t do! There are far more things you can do to optimize proxy processing speed. For a look at them, check out Part Two.