Vault secrets script client inc new 'keyring' client (#27669)

This adds a new type of vault-password script  (a 'client') that takes advantage of and enhances the 
multiple vault password support.

If a vault password script basename ends with the name '-client', consider it a vault password script client. 

A vault password script 'client' just means that the script will take a '--vault-id' command line arg.

The previous vault password script (as invoked by --vault-password-file pointing to an executable) takes
no args and returns the password on stdout. But it doesnt know anything about --vault-id or multiple vault
passwords.

The new 'protocol' of the vault password script takes a cli arg ('--vault-id') so that it can lookup that specific
vault-id and return it's password.

Since existing vault password scripts don't know the new 'protocol', a way to distinguish password scripts
that do understand the protocol was needed.  The convention now is to consider password scripts that are
named like 'something-client.py' (and executable) to be vault password client scripts.

The new client scripts get invoked with the '--vault-id' they were requested for. An example:

     ansible-playbook --vault-id my_vault_id@contrib/vault/vault-keyring-client.py some_playbook.yml

That will cause the 'contrib/vault/vault-keyring-client.py' script to be invoked as:

     contrib/vault/vault-keyring-client.py --vault-id my_vault_id

The previous vault-keyring.py password script was extended to become vault-keyring-client.py. It uses
the python 'keyring' module to request secrets from various backends. The plain 'vault-keyring.py' script
would determine which key id and keyring name to use based on values that had to be set in ansible.cfg.
So it was also limited to one keyring name.

The new vault-keyring-client.py will request the secret for the vault id provided via the '--vault-id' option.
The script can be used without config and can be used for multiple keyring ids (and keyrings).

On success, a vault password client script will print the password to stdout and exit with a return code of 0.
If the 'client' script can't find a secret for the --vault-id, the script will exit with return code of 2 and print an error to stderr.

contrib/vault/vault-keyring-client.py

@@ -0,0 +1,147 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# (c) 2014, Matt Martz <matt@sivel.net>
+# (c) 2016, Justin Mayer <https://justinmayer.com/>
+# This file is part of Ansible.
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+# =============================================================================
+#
+# This script is to be used with ansible-vault's --vault-id arg
+# to retrieve the vault password via your OS's native keyring application.
+#
+# This file *MUST* be saved with executable permissions. Otherwise, Ansible
+# will try to parse as a password file and display: "ERROR! Decryption failed"
+#
+# The `keyring` Python module is required: https://pypi.python.org/pypi/keyring
+#
+# By default, this script will store the specified password in the keyring of
+# the user that invokes the script. To specify a user keyring, add a [vault]
+# section to your ansible.cfg file with a 'username' option. Example:
+#
+# [vault]
+# username = 'ansible-vault'
+#
+# In useage like:
+#
+#    ansible-vault --vault-id keyring_id@contrib/vault/vault-keyring-client.py view some_encrypted_file
+#
+#  --vault-id will call this script like:
+#
+#     contrib/vault/vault-keyring-client.py --vault-id keyring_id
+#
+# That will retrieve the password from users keyring for the
+# keyring service 'keyring_id'. The equilivent of:
+#
+#      keyring get keyring_id $USER
+#
+# If no vault-id name is specified to ansible command line, the vault-keyring-client.py
+# script will be called without a '--vault-id' and will default to the keyring service 'ansible'
+# This is equilivent to:
+#
+#    keyring get ansible $USER
+#
+# You can configure the `vault_password_file` option in ansible.cfg:
+#
+# [defaults]
+# ...
+# vault_password_file = /path/to/vault-keyring-client.py
+# ...
+#
+# To set your password, `cd` to your project directory and run:
+#
+#   # will use default keyring service / vault-id of 'ansible'
+#   /path/to/vault-keyring-client.py --set
+#
+# or to specify the keyring service / vault-id of 'my_ansible_secret':
+#
+#  /path/to/vault-keyring-client.py --vault-id my_ansible_secret --set
+#
+# If you choose not to configure the path to `vault_password_file` in
+# ansible.cfg, your `ansible-playbook` command might look like:
+#
+# ansible-playbook --vault-id=keyring_id@/path/to/vault-keyring-client.py site.yml
+
+ANSIBLE_METADATA = {'status': ['preview'],
+                    'supported_by': 'community',
+                    'version': '1.0'}
+
+import argparse
+import sys
+import getpass
+import keyring
+
+from ansible.config.manager import ConfigManager
+
+KEYNAME_UNKNOWN_RC = 2
+
+
+def build_arg_parser():
+    parser = argparse.ArgumentParser(description='Get a vault password from user keyring')
+
+    parser.add_argument('--vault-id', action='store', default=None,
+                        dest='vault_id',
+                        help='name of the vault secret to get from keyring')
+    parser.add_argument('--username', action='store', default=None,
+                        help='the username whose keyring is queried')
+    parser.add_argument('--set', action='store_true', default=False,
+                        dest='set_password',
+                        help='set the password instead of getting it')
+    return parser
+
+
+def main():
+    config_manager = ConfigManager()
+    username = config_manager.data.get_setting('vault.username')
+    if not username:
+        username = getpass.getuser()
+
+    keyname = config_manager.data.get_setting('vault.keyname')
+    if not keyname:
+        keyname = 'ansible'
+
+    arg_parser = build_arg_parser()
+    args = arg_parser.parse_args()
+
+    username = args.username or username
+    keyname = args.vault_id or keyname
+
+    # print('username: %s keyname: %s' % (username, keyname))
+
+    if args.set_password:
+        intro = 'Storing password in "{}" user keyring using key name: {}\n'
+        sys.stdout.write(intro.format(username, keyname))
+        password = getpass.getpass()
+        confirm = getpass.getpass('Confirm password: ')
+        if password == confirm:
+            keyring.set_password(keyname, username, password)
+        else:
+            sys.stderr.write('Passwords do not match\n')
+            sys.exit(1)
+    else:
+        secret = keyring.get_password(keyname, username)
+        if secret is None:
+            sys.stderr.write('vault-keyring-client could not find key="%s" for user="%s" via backend="%s"\n' %
+                             (keyname, username, keyring.get_keyring().name))
+            sys.exit(KEYNAME_UNKNOWN_RC)
+
+        # print('secret: %s' % secret)
+        sys.stdout.write('%s\n' % secret)
+
+    sys.exit(0)
+
+
+if __name__ == '__main__':
+    main()

lib/ansible/cli/__init__.py

@@ -293,7 +293,7 @@ class CLI(with_metaclass(ABCMeta, object)):
             display.vvvvv('Reading vault password file: %s' % vault_id_value)
             # read vault_pass from a file
             file_vault_secret = get_file_vault_secret(filename=vault_id_value,
-                                                      vault_id_name=vault_id_name,
+                                                      vault_id=vault_id_name,
                                                       loader=loader)
 
             # an invalid password file will error globally

lib/ansible/parsing/vault/__init__.py

@@ -303,14 +303,33 @@ class PromptVaultSecret(VaultSecret):
             raise AnsibleError("Passwords do not match")
 
 
-def get_file_vault_secret(filename=None, vault_id_name=None, encoding=None, loader=None):
+def script_is_client(filename):
+    '''Determine if a vault secret script is a client script that can be given --vault-id args'''
+
+    # if password script is 'something-client' or 'something-client.[sh|py|rb|etc]'
+    # script_name can still have '.' or could be entire filename if there is no ext
+    script_name, dummy = os.path.splitext(filename)
+
+    # TODO: for now, this is entirely based on filename
+    if script_name.endswith('-client'):
+        return True
+
+    return False
+
+
+def get_file_vault_secret(filename=None, vault_id=None, encoding=None, loader=None):
     this_path = os.path.realpath(os.path.expanduser(filename))
 
     if not os.path.exists(this_path):
         raise AnsibleError("The vault password file %s was not found" % this_path)
 
     if loader.is_executable(this_path):
+        if script_is_client(filename):
+            display.vvvv('The vault password file %s is a client script.' % filename)
             # TODO: pass vault_id_name to script via cli
+            return ClientScriptVaultSecret(filename=this_path, vault_id=vault_id,
+                                           encoding=encoding, loader=loader)
+        # just a plain vault password script. No args, returns a byte array
         return ScriptVaultSecret(filename=this_path, encoding=encoding, loader=loader)
 
     return FileVaultSecret(filename=this_path, encoding=encoding, loader=loader)
@@ -370,25 +389,89 @@ class ScriptVaultSecret(FileVaultSecret):
         if not self.loader.is_executable(filename):
             raise AnsibleVaultError("The vault password script %s was not executable" % filename)
 
+        command = self._build_command()
+
+        stdout, stderr, p = self._run(command)
+
+        self._check_results(stdout, stderr, p)
+
+        vault_pass = stdout.strip(b'\r\n')
+
+        empty_password_msg = 'Invalid vault password was provided from script (%s)' % filename
+        verify_secret_is_not_empty(vault_pass,
+                                   msg=empty_password_msg)
+
+        return vault_pass
+
+    def _run(self, command):
         try:
             # STDERR not captured to make it easier for users to prompt for input in their scripts
-            p = subprocess.Popen(filename, stdout=subprocess.PIPE)
+            p = subprocess.Popen(command, stdout=subprocess.PIPE)
         except OSError as e:
             msg_format = "Problem running vault password script %s (%s)." \
                 " If this is not a script, remove the executable bit from the file."
-            msg = msg_format % (filename, e)
+            msg = msg_format % (self.filename, e)
 
             raise AnsibleError(msg)
 
         stdout, stderr = p.communicate()
+        return stdout, stderr, p
 
-        if p.returncode != 0:
-            raise AnsibleError("Vault password script %s returned non-zero (%s): %s" % (filename, p.returncode, stderr))
+    def _check_results(self, stdout, stderr, popen):
+        if popen.returncode != 0:
+            raise AnsibleError("Vault password script %s returned non-zero (%s): %s" %
+                               (self.filename, popen.returncode, stderr))
 
-        vault_pass = stdout.strip(b'\r\n')
-        verify_secret_is_not_empty(vault_pass,
-                                   msg='Invalid vault password was provided from script (%s)' % filename)
-        return vault_pass
+    def _build_command(self):
+        return [self.filename]
+
+
+class ClientScriptVaultSecret(ScriptVaultSecret):
+    VAULT_ID_UNKNOWN_RC = 2
+
+    def __init__(self, filename=None, encoding=None, loader=None, vault_id=None):
+        super(ClientScriptVaultSecret, self).__init__(filename=filename,
+                                                      encoding=encoding,
+                                                      loader=loader)
+        self._vault_id = vault_id
+        display.vvvv('Executing vault password client script: %s --vault-id=%s' % (filename, vault_id))
+
+    def _run(self, command):
+        try:
+            p = subprocess.Popen(command,
+                                 stdout=subprocess.PIPE,
+                                 stderr=subprocess.PIPE)
+        except OSError as e:
+            msg_format = "Problem running vault password client script %s (%s)." \
+                " If this is not a script, remove the executable bit from the file."
+            msg = msg_format % (self.filename, e)
+
+            raise AnsibleError(msg)
+
+        stdout, stderr = p.communicate()
+        return stdout, stderr, p
+
+    def _check_results(self, stdout, stderr, popen):
+        if popen.returncode == self.VAULT_ID_UNKNOWN_RC:
+            raise AnsibleError('Vault password client script %s did not find a secret for vault-id=%s: %s' %
+                               (self.filename, self._vault_id, stderr))
+
+        if popen.returncode != 0:
+            raise AnsibleError("Vault password client script %s returned non-zero (%s) when getting secret for vault-id=%s: %s" %
+                               (self.filename, popen.returncode, self._vault_id, stderr))
+
+    def _build_command(self):
+        command = [self.filename]
+        if self._vault_id:
+            command.extend(['--vault-id', self._vault_id])
+
+        return command
+
+    def __repr__(self):
+        if self.filename:
+            return "%s(filename='%s', vault_id='%s')" % \
+                (self.__class__.__name__, self.filename, self._vault_id)
+        return "%s()" % (self.__class__.__name__)
 
 
 def match_secrets(secrets, target_vault_ids):

test/integration/targets/vault/runme.sh

@@ -24,6 +24,14 @@ FORMAT_1_1_HEADER="\$ANSIBLE_VAULT;1.1;AES256"
 FORMAT_1_2_HEADER="\$ANSIBLE_VAULT;1.2;AES256"
 
 VAULT_PASSWORD_FILE=vault-password
+# new format, view, using password client script
+ansible-vault view "$@" --vault-id vault-password@test-vault-client.py format_1_1_AES256.yml
+
+# view, using password client script, unknown vault/keyname
+ansible-vault view "$@" --vault-id some_unknown_vault_id@test-vault-client.py format_1_1_AES256.yml && :
+WRONG_RC=$?
+echo "rc was $WRONG_RC (1 is expected)"
+[ $WRONG_RC -eq 1 ]
 
 # old format
 ansible-vault view "$@" --vault-password-file vault-password-ansible format_1_0_AES.yml

test/integration/targets/vault/test-vault-client.py

@@ -0,0 +1,63 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+ANSIBLE_METADATA = {'status': ['preview'],
+                    'supported_by': 'community',
+                    'version': '1.0'}
+
+import argparse
+import sys
+
+# TODO: could read these from the files I suppose...
+secrets = {'vault-password': 'test-vault-password',
+           'vault-password-wrong': 'hunter42',
+           'vault-password-ansible': 'ansible',
+           'password': 'password',
+           'vault-client-password-1': 'password-1',
+           'vault-client-password-2': 'password-2'}
+
+
+def build_arg_parser():
+    parser = argparse.ArgumentParser(description='Get a vault password from user keyring')
+
+    parser.add_argument('--vault-id', action='store', default=None,
+                        dest='vault_id',
+                        help='name of the vault secret to get from keyring')
+    parser.add_argument('--username', action='store', default=None,
+                        help='the username whose keyring is queried')
+    parser.add_argument('--set', action='store_true', default=False,
+                        dest='set_password',
+                        help='set the password instead of getting it')
+    return parser
+
+
+def get_secret(keyname):
+    return secrets.get(keyname, None)
+
+
+def main():
+    rc = 0
+
+    arg_parser = build_arg_parser()
+    args = arg_parser.parse_args()
+    # print('args: %s' % args)
+
+    keyname = args.vault_id or 'ansible'
+
+    if args.set_password:
+        print('--set is not supported yet')
+        sys.exit(1)
+
+    secret = get_secret(keyname)
+    if secret is None:
+        sys.stderr.write('test-vault-client could not find key for vault-id="%s"\n' % keyname)
+        # key not found rc=2
+        return 2
+
+    sys.stdout.write('%s\n' % secret)
+
+    return rc
+
+
+if __name__ == '__main__':
+    sys.exit(main())

test/units/parsing/vault/test_vault.py

@@ -233,6 +233,43 @@ class TestScriptVaultSecret(unittest.TestCase):
                                     secret.load)
 
 
+class TestScriptIsClient(unittest.TestCase):
+    def test_randomname(self):
+        filename = 'randomname'
+        res = vault.script_is_client(filename)
+        self.assertFalse(res)
+
+    def test_something_dash_client(self):
+        filename = 'something-client'
+        res = vault.script_is_client(filename)
+        self.assertTrue(res)
+
+    def test_something_dash_client_somethingelse(self):
+        filename = 'something-client-somethingelse'
+        res = vault.script_is_client(filename)
+        self.assertFalse(res)
+
+    def test_something_dash_client_py(self):
+        filename = 'something-client.py'
+        res = vault.script_is_client(filename)
+        self.assertTrue(res)
+
+    def test_full_path_something_dash_client_py(self):
+        filename = '/foo/bar/something-client.py'
+        res = vault.script_is_client(filename)
+        self.assertTrue(res)
+
+    def test_full_path_something_dash_client(self):
+        filename = '/foo/bar/something-client'
+        res = vault.script_is_client(filename)
+        self.assertTrue(res)
+
+    def test_full_path_something_dash_client_in_dir(self):
+        filename = '/foo/bar/something-client/but/not/filename'
+        res = vault.script_is_client(filename)
+        self.assertFalse(res)
+
+
 class TestGetFileVaultSecret(unittest.TestCase):
     def test_file(self):
         password = 'some password'