Tutorial: How to Get Google Webmaster Tools Data on Windows With Python

Tutorial: How to Get Google Webmaster Tools Data on Windows With Python

The Search Query report in Google Webmaster tools is more important than ever, with the ominous (not provided) mask hiding 25%-40% of referring keyword traffic in Google Analytics. Google recently made WMT data available through an open source Python Library, making it easy to transfer that data straight into Google Docs or to your desktop, but setup and configuration isn’t easy for most.

To make it a little easier to access that data, I’ll walk you through the beginners steps to get search queries from Google Webmaster Tools using the Google Data Python Library on Windows. Expect this to take at least 15 minutes.

1. Download and install Python

Python LogoFirst, download the Windows Installer from the Python.org official download page. At the time of writing, you’ll want the “Python 2.7.2 Windows Installer (Windows binary — does not include source).” If you have a 64-bit machine, make sure you select the “X86-64 Installer.” Once downloaded, open it up and follow the installation wizard. I recommend choosing default settings if you’re a beginner but feel free to customize if you know what you’re doing.

Installing with default settings will install Python in the C:Python27 directory. You don’t need to touch anything in there yet.

2. Download and extract the Google Data APIs Client Library for Python.

Download the latest Google Data API Client Library in .zip or .tar.gz format (recommended if you use WinRar or 7zip), depending on your preference, and extract the contents to the C:Python27 folder.

Right Click and Select Extract All

Extract Destination settings should extract the file to the Python27 folder

 

That will place the zipped “gdata-2.0.16″ folder in your C:Python27 folder, as follows: C:Python27gdata-2.0.16 , which should contain the following contents:

Extract results will have these contents

IMPORTANT: Now, you need to copy ALL the contents of the gdata-2.0.16 folder and paste them straight into the Python27 folder. You will probably get a popup saying that the README file already exists – you can  replace it, or skip it, you don’t really need either unless you enjoy the literary merits of README files.

Read more about the Google Data APIs Python Client Library.

3. Install the Google Data API Client Library

Now it gets fun – it’s time to fire up your Windows Command Prompt by finding it in your “Accessories” folder, or by pressing Windows+R, typing “cmd” and hitting “Enter.”

The Windows Command Prompt is in the Accessories folder

  • Navigate into the C:Python27 folder by typing “cd Python27” without quotes.
    “cd” is short for “change directory,” which is how to move from one folder (directory) to another from the command line.
    Use the cd command to move to the python27 folder in the root C directory
  • Now that you’re in the Python27 folder, type “python ./setup.py install” without quotes, which will install the client library. If you start seeing hundreds of scrolling lines, you’ve done everything correctly to this point.
    Type the command above to install the Data API Script
  • (optional) Now you can start playing around with the sample scripts that came with the Data API Library – here is some information on how to do that.

4. Create a ‘wmt’ folder, and set up scripts

  • Create a folder in the Python27 folder called wmt, so that it can be found at C:Python27wmt
  • Download and place the downloader.py script in the new wmt folder (either copy the code from that page into a new file and save it as downloader.py, or right click the “view raw file” link on that page and save it to the folder).
  • Download and place the example-create-spreadsheet.py script to the wmt folder, and right click the file and select “Edit with IDLE”
Right click and select Edit with IDLE
  • Find the highlighted fields and replace them with your Webmaster Tools user credentials and website, then save (Ctrl+s):
    • email
    • password
    • website
Edit these values

5. Open the command line and start getting Search Query reports

Now that C:Python27wmtexample-create-spreadsheet.py has been updated with your website of choice, along with your Webmaster Tools username and password, open your Command Line prompt, (make sure you’re still working in the C:Python27 directory), and type in this command:

  • python ./wmt/example-create-spreadsheet.py

After several seconds you will see a message that says “Spreadsheet now accessible online at: https://docs.google.com/spreadsheet/….,” and you can access the CSV file at that URL or by logging into Google Docs!

That’s it!

If you would rather download it directly to your desktop rather than transferring the data to Google Docs, use example-simple-download.py instead.

6. Handy tip

The example-create-spreadsheet.py code provided for Google is great if you consistently pull search query data from only one website, but if you’re like me and you need to easily pull reports for multiple websites, I recommend downloading my modified version:  sq-report.py (right click->Save as…). Everything is the same except for the website field, which you can now call from the command line like so:

python ./wmt/sq-report.py http://www.seo.com/

Custom Script

Replace http://www.seo.com/ with any website your Webmaster Tools account has access to – and don’t forget to edit the sq-report.py file with your username and password.

Now you can easily pull a report for any site you have access to, without editing the Python file every time!

7. Conclusion

I will follow up with instructions for automating this every month for compiling historic data, as Google Webmaster Tools only provides the past 30 days of data.

I hope this is helpful for the average Windows user who may find it trick to install and work with Python code, let me know how it goes and if you run into problems, contact me on Twitter or in the comments so I can help you troubleshoot.

Get Internet Marketing Insight For Your Company - SEO.com

5 Comments

  1. Haylee Andrews says

    Ive been doing seo for a long time and ive never considered this. Why would you use Python instead of just downloading it from webmaster tools? Really cool post!

  2. Mark Ginsberg says

    Hey Claye,

    Thanks for the walk through. It was very easy to follow and I got it working in no time. One quick question – is there a similar method to use to pull all the crawl errors on the site – I’ve been searching and haven’t found any guides on how to do this, except for Vanessa Fox’s article on Search Englie Land, which says you can do this but doesn’t tell you how. It would be awesome if you could point me in the right direction.

    Thanks 

  3. Claye Stokes says

    That would be a great subject for another post in this series – I’ll put it together, though I may rely on a method to get that from the API rather than a Python route.

    @Ignite Visibility woah, sorry for the belated reply, didn’t see your comment. Automation, my friend, vs logging in and clicking around, is the main reason I prefer Python.

    Same idea as using Ctrl+C/V for copy/paste, rather than clicking Edit->Copy/Paste with a mouse. Good luck!

  4. Marius says

    Hy! I saw that you have not accepted my first comment. Please, if you have some time, help me. I did as you written in the article and in the end I get this error:
    AttributeError: ‘function’ object has no attribute ‘func_name’
    Thank you!

Leave a Reply