Setting Up for Proxied SERP Requests

Whatever the sphere of business operation, we all need up-to-date information – lots of it, as quickly as we can get it. This calls for powerful and flexible solutions that can do the heavy lifting for us. And proxy servers can be the solution. You need dependable, high-quality proxy servers. Setting up for proxied SERP requests can help you save money and rapidly generate leads.

This tutorial focuses on setting up your device for SERP extraction. Search Engine Results Pages are web pages displayed when users search online with a search engine like Google. You may query specific terms or keywords. This helps you find the highest-ranking pages in search results, and market to those pages. And you can find high-ranking pages to do business with.

This tutorial shows you how to set up your computer for SERP projects. These steps include:

Optimization for personal browser and OS (a safeguard against blocking).
System settings
Browser settings
Setup of proxy in Windows 10
Setup of proxy in MacOS

You can also test your settings in a browser.

If you plan to use a rank-tracking application in conjunction with the proxy, you’ll need to consult the manufacturer’s website for any additional configuration or settings.

Simplifying SEO

Use cases

Some use cases which will benefit from integration of a proxy and a rank-tracking application are:

Researching your competition

Good SEO proxies provide you with the tools to check out the competition’s setup and successful strategies anonymously, and gain insight into how they operate.

Social media marketing

Social media marketing is a significant force for increasing your visibility and brand popularity.

Scraping the web

Information – a crucial part of an enterprise’s growth strategy – is readily available on the Internet, but the amount of time spent manually gathering that data would be prohibitive. The web scraping process saves time by allowing for automated data extraction.

Before you set up: Notes on optimization

Once you’ve determined which search engine you’ll scrape, take thought for the settings below.

URLs

If you’re querying Google, you can only query the location where the proxy server is based. For example, query the US-based Google – google.com – with a US-based proxy. Other search engines may have variations on this rule.
Generally, while you can connect to a proxy over HTTP, you should send queries to a search engine over HTTPS.
Make the URLs in your query look “organic” – that is, as if you just came across them in search results and did not try to elicit them specifically. This is especially important for Google searches.

Connections

Close the TCP/IP connection after completion of each unique keyword query.
After each unique keyword request, discard cookies.

User Agents

Rotation of your user agents can also be helpful in web scraping. Recurrent requests from the same IP address arouse suspicion, and the IP address may be blocked. Use only current versions of user agents for browser and operating systems. Check them every 6 months and replace them if necessary. Older user agents are easily detected and blocked.

Employ user agents randomly but retain the same user agent for each SERP for each keyword.

Be adaptable

Note that Google and other search engines can change their algorithms and behavior at any time. So, you may at times need to adjust your scraping technique or how your software operation.

Generally, these rules of thumb work well for SERP on Google and other search engines. For additional discussion and variations among other engines, please see 18 Alternative Search Engines to Google (You Should Also Optimize Your Website for).

System and browser settings

Important things to remember about these settings:

System settings determine how your device processes all of your Internet traffic. When system proxy settings are turned on, all data sent via the Internet is routed through the proxy server.
Browser settings send your traffic via the proxy server only when you’re using the browser to surf the web or download items. Any updates to browser proxy settings affect only the browser environment. Other data to and from the Internet will not go through the proxy server.
Proxy server settings will remain in place when you close the browser. After using the proxy server, you’ll need to tell the browser to disconnect from the proxy. Otherwise, the next time you use the browser, your traffic will continue to be routed through the proxy server.

Set up a proxy server in Windows 10

Authentication methods

You can authenticate either through an authorized static IP address list or by username:password.

While authorizing static IP addresses is the more secure method, you may prefer to include username:password in your scripts.

IP Authentication

With the search bar enabled, enter proxy server settings. Click on the first result to display the settings window.

OR:

Press Windows + I together to open the main Settings menu and search for proxy server settings.

Confirm that the Automatically detect settings toggle is set to On. Under Manual Setup, set the Use a Proxy Server toggle to On.
Using data from your proxy service provider:
- Enter the proxy server IP address or hostname in the Address field.
- Enter the port number in the Port field (supplied by your provider).
Click on Save.
If you need to enter a username and password to connect to the proxy server, please see the following section on Basic Authentication.

Basic Authentication

With the search bar enabled, type credential manager. Click on the first result to open the credential manager settings window.

OR:

Press Windows + I together to open the main Settings menu and search for credential manager.

Select Windows Credentials and click on Add a Windows credential.

Enter the proxy server IP address or hostname in the Internet or network address field. Using data from your provider, enter your username and password in the corresponding fields.
Click OK.

This concludes Windows 10 setup. Hereafter, traffic sent to the Internet goes through the proxy server.

Set up a proxy server in MacOS

Click on the Apple menu and then System Preferences.
Next, click on the Network icon and select your connection, either Wi-Fi or Ethernet.
Now click Advanced and then Proxies.

Click on the Apple menu and then System Preferences.
Next, click on the Network icon and select your connection, either Wi-Fi or Ethernet.
Now click Advanced and then Proxies.

This concludes the MacOS setup.

Set up a proxy in Chrome

Chrome and other Chromium-based browsers use your Windows or MacOS system settings for proxy servers. When setting up a proxy server in Chrome, go through browser settings and then into your device’s operating system.

Follow the steps below to change your proxy settings from inside Chrome:

Open the Chrome toolbar by clicking on the three dots at the top right of the browser and from the menu click on Settings.
In the search bar at the top of the settings, type in proxy. Chrome automatically runs the search for you.

Click on Open your computer’s proxy settings. Chrome opens your system settings.
Now return to the instructions for your operating system, either Windows or MacOS.
Complete the applicable instructions for OS setup.

This concludes the setup for Chrome browser. After you’ve completed applicable instructions for OS setup, traffic sent to the Internet will go through the proxy server.

Test settings in a browser

Before submitting queries in a live environment, best practice is to test them in a web browser to identify any authentication issues. After entering the proxy server details in your browser settings, go to your preferred specific Google ccTLDs, such as www.google.com. You will know settings are correct if you see the normal Google home page; then it’s OK to run queries.

If you will use a rank-tracking app in conjunction with the proxy, follow configuration steps for the tracker before sending requests.

Are you connected to the proxy?

Once you’ve authorized a proxy from your proxy service provider’s list, and confirmed that it’s online and available for use, you can confirm whether you’re connected to the proxy this way:

Go to a site such as http://ip.pizza/
See if the site is still showing your local IP address or if it has changed to another IP.
If connected to another IP, check it against the address of the proxy you’ve selected or had assigned.
You are not connected to the proxy if the site displays your own IP address.
Not connected? Then:
- Make sure you’ve authorized/whitelisted the assigned proxy.
- Wait a couple of minutes and try again.

Wrap up

Search engine results can change rapidly. Having the latest data from SERP rankings helps to tell you where your business stands in relation to competitors. And that can help you implement a timely, effective business strategy.