The Search Query report in Google Webmaster tools is more important than ever, with the ominous (not provided) mask hiding 25%-40% of referring keyword traffic in Google Analytics. Google recently made WMT data available through an open source Python Library, making it easy to transfer that data straight into Google Docs or to your desktop, but setup and configuration isn't easy for most.
To make it a little easier to access that data, I'll walk you through the beginners steps to get search queries from Google Webmaster Tools using the Google Data Python Library on Windows. Expect this to take at least 15 minutes.
1. Download and install Python
First, download the Windows Installer from the Python.org official download page. At the time of writing, you'll want the "Python 2.7.2 Windows Installer (Windows binary -- does not include source)." If you have a 64-bit machine, make sure you select the "X86-64 Installer." Once downloaded, open it up and follow the installation wizard. I recommend choosing default settings if you're a beginner but feel free to customize if you know what you're doing.
Installing with default settings will install Python in the C:Python27 directory. You don't need to touch anything in there yet.
2. Download and extract the Google Data APIs Client Library for Python.
Download the latest Google Data API Client Library in .zip or .tar.gz format (recommended if you use WinRar or 7zip), depending on your preference, and extract the contents to the C:Python27 folder.
That will place the zipped "gdata-2.0.16" folder in your C:Python27 folder, as follows: C:Python27gdata-2.0.16 , which should contain the following contents:
IMPORTANT: Now, you need to copy ALL the contents of the gdata-2.0.16 folder and paste them straight into the Python27 folder. You will probably get a popup saying that the README file already exists - you can replace it, or skip it, you don't really need either unless you enjoy the literary merits of README files.
Read more about the Google Data APIs Python Client Library.
3. Install the Google Data API Client Library
Now it gets fun - it's time to fire up your Windows Command Prompt by finding it in your "Accessories" folder, or by pressing Windows+R, typing "cmd" and hitting "Enter."
- Navigate into the C:Python27 folder by typing "cd Python27" without quotes.
"cd" is short for "change directory," which is how to move from one folder (directory) to another from the command line.
- Now that you're in the Python27 folder, type "python ./setup.py install" without quotes, which will install the client library. If you start seeing hundreds of scrolling lines, you've done everything correctly to this point.
- (optional) Now you can start playing around with the sample scripts that came with the Data API Library - here is some information on how to do that.
4. Create a 'wmt' folder, and set up scripts
- Create a folder in the Python27 folder called wmt, so that it can be found at C:Python27wmt
- Download and place the downloader.py script in the new wmt folder (either copy the code from that page into a new file and save it as downloader.py, or right click the "view raw file" link on that page and save it to the folder).
- Download and place the example-create-spreadsheet.py script to the wmt folder, and right click the file and select "Edit with IDLE"
- Find the highlighted fields and replace them with your Webmaster Tools user credentials and website, then save (Ctrl+s):
5. Open the command line and start getting Search Query reports
Now that C:Python27wmtexample-create-spreadsheet.py has been updated with your website of choice, along with your Webmaster Tools username and password, open your Command Line prompt, (make sure you're still working in the C:Python27 directory), and type in this command:
- python ./wmt/example-create-spreadsheet.py
After several seconds you will see a message that says "Spreadsheet now accessible online at: https://docs.google.com/spreadsheet/....," and you can access the CSV file at that URL or by logging into Google Docs!
If you would rather download it directly to your desktop rather than transferring the data to Google Docs, use example-simple-download.py instead.
6. Handy tip
The example-create-spreadsheet.py code provided for Google is great if you consistently pull search query data from only one website, but if you're like me and you need to easily pull reports for multiple websites, I recommend downloading my modified version: sq-report.py (right click->Save as...). Everything is the same except for the website field, which you can now call from the command line like so:
python ./wmt/sq-report.py http://www.seo.com/
Replace http://www.seo.com/ with any website your Webmaster Tools account has access to - and don't forget to edit the sq-report.py file with your username and password.
Now you can easily pull a report for any site you have access to, without editing the Python file every time!
I will follow up with instructions for automating this every month for compiling historic data, as Google Webmaster Tools only provides the past 30 days of data.
I hope this is helpful for the average Windows user who may find it trick to install and work with Python code, let me know how it goes and if you run into problems, contact me on Twitter or in the comments so I can help you troubleshoot.