Update GCE module to use JSON credentials (#13623)

* Update GCE module to use JSON credentials

* Ensure minimum libcloud version when using JSON crednetials for GCE

* Relax langauge around libcloud requirements

docsite/rst/guide_gce.rst

@@ -11,7 +11,7 @@ Introduction
 Ansible contains modules for managing Google Compute Engine resources, including creating instances, controlling network access, working with persistent disks, and managing
 load balancers.  Additionally, there is an inventory plugin that can automatically suck down all of your GCE instances into Ansible dynamic inventory, and create groups by tag and other properties.
 
-The GCE modules all require the apache-libcloud module, which you can install from pip:
+The GCE modules all require the apache-libcloud module which you can install from pip:
 
 .. code-block:: bash
 
@@ -22,16 +22,19 @@ The GCE modules all require the apache-libcloud module, which you can install fr
 Credentials
 -----------
 
-To work with the GCE modules, you'll first need to get some credentials. You can create new one from the `console <https://console.developers.google.com/>`_ by going to the "APIs and Auth" section and choosing to create a new client ID for a service account. Once you've created a new client ID and downloaded (you must click **Generate new P12 Key**) the generated private key (in the `pkcs12 format <http://en.wikipedia.org/wiki/PKCS_12>`_), you'll need to convert the key by running the following command:
+To work with the GCE modules, you'll first need to get some credentials in the
+JSON format:
 
-.. code-block:: bash
+1. `Create a Service Account <https://developers.google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount>`_
+2. `Download JSON credentials <https://support.google.com/cloud/answer/6158849?hl=en&ref_topic=6262490#serviceaccounts>`_
 
-    $ openssl pkcs12 -in pkey.pkcs12 -passin pass:notasecret -nodes -nocerts | openssl rsa -out pkey.pem
+There are three different ways to provide credentials to Ansible so that it can talk with Google Cloud for provisioning and configuration actions:
 
-There are two different ways to provide credentials to Ansible so that it can talk with Google Cloud for provisioning and configuration actions:
+.. note:: If you would like to use JSON credentials you must have libcloud >= 0.17.0
 
 * by providing to the modules directly
 * by populating a ``secrets.py`` file
+* by setting environment variables
 
 Calling Modules By Passing Credentials
 ``````````````````````````````````````
@@ -39,7 +42,7 @@ Calling Modules By Passing Credentials
 For the GCE modules you can specify the credentials as arguments:
 
 * ``service_account_email``: email associated with the project
-* ``pem_file``: path to the pem file
+* ``credentials_file``: path to the JSON credentials file
 * ``project_id``: id of the project
 
 For example, to create a new instance using the cloud module, you can use the following configuration:
@@ -53,7 +56,7 @@ For example, to create a new instance using the cloud module, you can use the fo
 
      vars:
        service_account_email: unique-id@developer.gserviceaccount.com
-       pem_file: /path/to/project.pem
+       credentials_file: /path/to/project.json
        project_id: project-id
        machine_type: n1-standard-1
        image: debian-7
@@ -66,23 +69,45 @@ For example, to create a new instance using the cloud module, you can use the fo
             machine_type: "{{ machine_type }}"
             image: "{{ image }}"
             service_account_email: "{{ service_account_email }}"
-            pem_file: "{{ pem_file }}" 
+            credentials_file: "{{ credentials_file }}"
             project_id: "{{ project_id }}"
 
-Calling Modules with secrets.py
+When running Ansible inside a GCE VM you can use the service account credentials from the local metadata server by
-```````````````````````````````
+setting both ``service_account_email`` and ``credentials_file`` to a blank string.
+
+Configuring Modules with secrets.py
+```````````````````````````````````
 
 Create a file ``secrets.py`` looking like following, and put it in some folder which is in your ``$PYTHONPATH``:
 
 .. code-block:: python
 
-    GCE_PARAMS = ('i...@project.googleusercontent.com', '/path/to/project.pem')
+    GCE_PARAMS = ('i...@project.googleusercontent.com', '/path/to/project.json')
     GCE_KEYWORD_PARAMS = {'project': 'project_id'}
 
 Ensure to enter the email address from the created services account and not the one from your main account.
 
 Now the modules can be used as above, but the account information can be omitted.
 
+If you are running Ansible from inside a GCE VM with an authorized service account you can set the email address and
+credentials path as follows so that get automatically picked up:
+
+.. code-block:: python
+
+    GCE_PARAMS = ('', '')
+    GCE_KEYWORD_PARAMS = {'project': 'project_id'}
+
+Configuring Modules with Environment Variables
+``````````````````````````````````````````````
+
+Set the following environment variables before running Ansible in order to configure your credentials:
+
+.. code-block:: bash
+
+    GCE_EMAIL
+    GCE_PROJECT
+    GCE_CREDENTIALS_FILE_PATH
+
 GCE Dynamic Inventory
 ---------------------
 
@@ -171,7 +196,7 @@ A playbook would looks like this:
        machine_type: n1-standard-1 # default
        image: debian-7
        service_account_email: unique-id@developer.gserviceaccount.com
-       pem_file: /path/to/project.pem
+       credentials_file: /path/to/project.json
        project_id: project-id
 
      tasks:
@@ -181,7 +206,7 @@ A playbook would looks like this:
              machine_type: "{{ machine_type }}"
              image: "{{ image }}"
              service_account_email: "{{ service_account_email }}"
-             pem_file: "{{ pem_file }}"
+             credentials_file: "{{ credentials_file }}"
              project_id: "{{ project_id }}"
              tags: webserver
          register: gce
@@ -224,7 +249,7 @@ a basic example of what is possible::
         machine_type: n1-standard-1 # default
         image: debian-7
         service_account_email: unique-id@developer.gserviceaccount.com
-        pem_file: /path/to/project.pem
+        credentials_file: /path/to/project.json
         project_id: project-id
 
       roles:
@@ -241,10 +266,9 @@ a basic example of what is possible::
             allowed: "tcp:80"
             state: "present"
             service_account_email: "{{ service_account_email }}"
-            pem_file: "{{ pem_file }}" 
+            credentials_file: "{{ credentials_file }}"
             project_id: "{{ project_id }}"
 
 By pointing your browser to the IP of the server, you should see a page welcoming you.
 
 Upgrades to this documentation are welcome, hit the github link at the top right of this page if you would like to make additions!
-

lib/ansible/module_utils/gce.py

@@ -27,10 +27,14 @@
 # USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 #
 
+import json
 import os
 import traceback
 
 from libcloud.compute.types import Provider
+from distutils.version import LooseVersion
+
+import libcloud
 from libcloud.compute.providers import get_driver
 
 USER_AGENT_PRODUCT="Ansible-gce"
@@ -39,6 +43,7 @@ USER_AGENT_VERSION="v1"
 def gce_connect(module, provider=None):
     """Return a Google Cloud Engine connection."""
     service_account_email = module.params.get('service_account_email', None)
+    credentials_file = module.params.get('credentials_file', None)
     pem_file = module.params.get('pem_file', None)
     project_id = module.params.get('project_id', None)
 
@@ -50,6 +55,8 @@ def gce_connect(module, provider=None):
         project_id = os.environ.get('GCE_PROJECT', None)
     if not pem_file:
         pem_file = os.environ.get('GCE_PEM_FILE_PATH', None)
+    if not credentials_file:
+        credentials_file = os.environ.get('GCE_CREDENTIALS_FILE_PATH', pem_file)
 
     # If we still don't have one or more of our credentials, attempt to
     # get the remaining values from the libcloud secrets file.
@@ -62,25 +69,41 @@ def gce_connect(module, provider=None):
         if hasattr(secrets, 'GCE_PARAMS'):
             if not service_account_email:
                 service_account_email = secrets.GCE_PARAMS[0]
-            if not pem_file:
+            if not credentials_file:
-                pem_file = secrets.GCE_PARAMS[1]
+                credentials_file = secrets.GCE_PARAMS[1]
         keyword_params = getattr(secrets, 'GCE_KEYWORD_PARAMS', {})
         if not project_id:
             project_id = keyword_params.get('project', None)
 
     # If we *still* don't have the credentials we need, then it's time to
     # just fail out.
-    if service_account_email is None or pem_file is None or project_id is None:
+    if service_account_email is None or credentials_file is None or project_id is None:
         module.fail_json(msg='Missing GCE connection parameters in libcloud '
                              'secrets file.')
         return None
+    else:
+        # We have credentials but lets make sure that if they are JSON we have the minimum
+        # libcloud requirement met
+        try:
+            # Try to read credentials as JSON
+            with open(credentials_file) as credentials:
+                json.loads(credentials.read())
+            # If the credentials are proper JSON and we do not have the minimum
+            # required libcloud version, bail out and return a descriptive error
+            if LooseVersion(libcloud.__version__) < '0.17.0':
+                module.fail_json(msg='Using JSON credentials but libcloud minimum version not met. '
+                                     'Upgrade to libcloud>=0.17.0.')
+                return None
+        except ValueError, e:
+            # Not JSON
+            pass
 
     # Allow for passing in libcloud Google DNS (e.g, Provider.GOOGLE)
     if provider is None:
         provider = Provider.GCE
 
     try:
-        gce = get_driver(provider)(service_account_email, pem_file,
+        gce = get_driver(provider)(service_account_email, credentials_file,
                 datacenter=module.params.get('zone', None),
                 project=project_id)
         gce.connection.user_agent_append("%s/%s" % (