Avoid resolving a pattern that is a plain host. When matching a hostname in the
hosts_cache, just use the host object from there.
When running a task on say 750 hosts, this yields a huge improvement.
In some cases, where a host is mentioned in multiple groups, and those
groups are referenced in multiple ini files, a group could still contain
multiple instances of a group in its host,groups list, where only one of them
is the right group, that exists in the inventory.
Split out parsing of vars files to per host and per group
parsing, instead of reparsing all groups for each host. This enhances
performance.
Extend vars_plugins' API with two new methods:
* get host variables: only parses host_vars
* get group variables: only parses group_vars for specific group
The initial run method is still used for backward compatibility.
Parse all vars_plugins at inventory initialisation, instead of
per host when touched first by runner. Here we can also loop through
all groups once easily, then parse them.
This also centralizes all parsing in the inventory constructor.
modified: bin/ansible
modified: bin/ansible-playbook
modified: lib/ansible/inventory/__init__.py
modified: lib/ansible/inventory/vars_plugins/group_vars.py
If something is executable but doesn't look like it should be, or if
something is NOT executable and DOES looks like it should, show a
more apropos error with a hint on correcting the problem
Fixes#5113
Make sure all hosts and groups are unique objects
and that those are referenced uniquely everywhere.
Also fixes test_dir_inventory unit tests which were broken after previous
patches.
modified: lib/ansible/inventory/dir.py
Fixes a bug where vault_password parameter was not passed through in
_load_vars_from_folder()
modified: lib/ansible/inventory/vars_plugins/group_vars.py
0. Uncomment the test.
1. Test fails.
2. Make vars unique per file in test inventory files.
3. Modify token addition to not ast.literal_eval(v) a variable containing a hash.
4. Modify vars to have an escape in test inventory file.
5. Catch exceptions explicitly. Any unknown exceptions should be a bug.
6. Test passes.
It came up that fixing this unit test may relate to another ticket that is open. This work allows us to uncomment this unit test by fixing how we pars variables allowing a quoted variable to contain a '#'.
Work also went into cleaning up some of the test data to clarify what was working.
Lastly work went into cleaning up formatting so that the code is easily read.
As part of 94f3b9bfab the code was changed to support dynamically adding localhost to the inventory. This change introduced an crash when run via ansible-pull
```
Starting ansible-pull at 2014-01-20 23:09:57
Traceback (most recent call last):
File "/tmp/ansible/bin/ansible", line 157, in <module>
(runner, results) = cli.run(options, args)
File "/tmp/ansible/bin/ansible", line 82, in run
hosts = inventory_manager.list_hosts(pattern)
File "/tmp/ansible/lib/ansible/inventory/__init__.py", line 372, in list_hosts
result = [ h.name for h in self.get_hosts(pattern) ]
File "/tmp/ansible/lib/ansible/inventory/__init__.py", line 136, in get_hosts
subset = self._get_hosts(self._subset)
File "/tmp/ansible/lib/ansible/inventory/__init__.py", line 177, in _get_hosts
that = self.__get_hosts(p)
File "/tmp/ansible/lib/ansible/inventory/__init__.py", line 198, in __get_hosts
hpat = self._hosts_in_unenumerated_pattern(name)
File "/tmp/ansible/lib/ansible/inventory/__init__.py", line 275, in _hosts_in_unenumerated_pattern
ungrouped.add_host(new_host)
AttributeError: 'NoneType' object has no attribute 'add_host'
```
The root cause is there is no group for the host to be added to. I fixed this case by creating the ungrouped group when it doesn't exist and then adding the host to the newly added group. This fixes the regression for me.
So instead of having:
group_vars/production.yml
A user could chose to reorganize to:
group_vars/production/staff.yml
group_vars/production/networks.yml
group_vars/production/dns.yml
(Backwards compatible.)
- Move all the supported YAML file extensions into a constant
- Use helper functions to avoid duplicate code for group/host vars
- Catch and disallow some confusing situations, such as the presence of
multiple group/host vars files for the same group/host, but with
different extensions. For example having both group_vars/all.yml and
group_vars/all.yaml.
- Catch and report file system permission issues, symlink errors,
unexpected file system objects
- Trivial performance improvement from making fewer stat system calls
- Restructuring that makes it easy for a following patch to support
directory recursion
For link-local addresses, it is sometimes necessary to append the
interface to use for the ipv6 address. This patch extends the ipv6
regex to allow for '%ifnameX' at the end.
See https://bugzilla.redhat.com/show_bug.cgi?id=136852 for more info
testing with a ipv6 :
ansible -u misc -i '[2002::c23e]:22,' '*' -m ping
fail due to parsing of ':' as a separator of port/ip with ipv4.
This commit add support for properly parsing 2002::c23 and the
bracket notation [2002::ce]:2222
to ensure consistent behavior, hosts should look like this:
hosts: webservers:&boston:!rack42
So when applying the host selectors, run those without the "&" first,
then the &s, then the !s.
Closes#3500
and the _meta hash contains a "hostvars", don't call --host hostname for any elements
and just serve them directly for performance enhancements with the external inventory
script and a large number of hosts.
Since ansible 1.2, it became possible to place a host_vars
directory in the same directory as a playbook, making it possible
to keep host_vars local to that playbook there. However, due to
python's os.path.dirname, a action such as:
$ ansible-playbook pb.yml
..would not pick up the host_vars as os.path.dirname("pb.yml")
returns "", unlike the unix command dirname that would return
".". Substituting "pb.yml" on the command line with "./pb.yml"
would do the trick, but is not always intuitive. This patch
solves the problem until python solves issue18547 [1].
[1] http://bugs.python.org/issue18547
A host pattern of the form '!foo' by itself does not work, but
'all:!foo' does. If the first pattern is a negation, this commit
automatically prepends 'all'.
Signed-off-by: martin f. krafft <madduck@madduck.net>
e.g. db[01:10:3]node-[01:10]
- to do this we split off at the first [...] set, getting the list
of hosts and then repeat until none left.
- also add an optional third parameter which contains the step. (Default: 1)
so range can be [01:10:2] -> 01 03 05 07 09
If someone has a " #" in a quoted var string, it
will interpret that as a comment and refuse to
load the inventory file due to an unbalanced
quote. Noisy failure > unexpected behavior.
Look for a file with the base name of the group/host, first without
a file extension, then with a '.yml' extension, and, finally, with
a '.yaml' extension, loading vars from only the first one found.
Hash variables are currently overriden if they are redefined. This
doesn't let the user refine hash entries or overriding selected keys,
which can, for some, be a desirable feature.
This patch let the user force hash merging by setting the
hash_behaviour value to "merge" (without the quotes) in ansible.cfg
However, by default, ansible behaves like it always did and if any value
besides "merge" is used ("replace" is suggested in the example ansible.cfg
file), it will also behave as always.
This allows patterns such as webservers:!debian:&datacenter1 to target
hosts in the webservers group, that are not in the debian group, but are
in the datacenter1 group. It also parses patterns left to right.
commit 48069adf0f47b09f675a9ba59360ca216b695ceb
Author: Gregory Duchatelet <skygreg@gmail.com>
Date: Tue Nov 27 10:13:08 2012 +0100
Removing this plugin from this branch.
commit 15400fffe643ad3e66d6b5a296fe62d36d9a617a
Author: Gregory Duchatelet <skygreg@gmail.com>
Date: Tue Nov 27 09:53:16 2012 +0100
Enhance _match function in inventory with regex.
--limit ~regex could be used to filter hosts or group with a regex.
Tested on cli and ansible-playbook.
commit 63c1b2e17ec6daa282e0a3af2249bda8f734dcd3
Author: Gregory Duchatelet <skygreg@gmail.com>
Date: Tue Nov 27 09:03:41 2012 +0100
Revert pull request #1684
commit 7c2c6fee3a856c52c1960b96ec2e7ca1112c35a1
Merge: f023a2f dd5a847
Author: Gregory Duchatelet <skygreg@gmail.com>
Date: Tue Nov 27 08:52:53 2012 +0100
Merge remote branch 'upstream/devel' into devel
commit f023a2f3df4a4c2ab2dfcc5aa42c57c02358a47a
Author: Gregory Duchatelet <skygreg@gmail.com>
Date: Mon Nov 26 20:52:27 2012 +0100
Add an inventory plugin to fetch groups and host from our CMDB.
commit c64193b4c67053e6e197b89c7143b9770cf71f23
Author: Gregory Duchatelet <skygreg@gmail.com>
Date: Mon Nov 26 20:43:30 2012 +0100
Added possibility to filter hosts from a group, with a regex, separating
groupname and regex with a ~
Usage in group pattern: group~filterpattern
Samples:
ansible group~server-0[1236] -m ping
ansible web~proxy -m ping
ansible web~(proxy|frontend) -m ping
Otherwise, a host in two groups, A and B, using a variable defined
in group A and all will get the value of all, as B's variables will
include the all variable.
Partially fixes#1647.
The default is 0, which means all hosts. If set to 1, each host would run a playbook all the way through before moving
on the next host. Fact gathering is still parallel, regardless of the serial setting.
to an additional pattern (a subset) specified on the command line. For instance, a playbook could be reusable
and target "webservers" and "dbservers", but you want to test only in the stage environment, or a few boxes at a time.
./group_vars/groupname.yml (for all groups the host is in)
./host_vars/hostname.yml (for the hostname given in the inventory)
This requires an actual inventory file, not script and the paths are relative
to the directory of the inventory file.
commit 4430ce3eefcdff0b0ceffea0ef66ea8e876a807d
Merge: 631783b 649963c
Author: Michael DeHaan <michael.dehaan@gmail.com>
Date: Thu Jul 12 01:28:43 2012 -0400
Merge branch 'host-groups' of https://github.com/dagwieers/ansible into daggroups
commit 649963ca2c2610b97a90d2449132a57ae6b39ec9
Author: Dag Wieërs <dag@wieers.com>
Date: Thu Jul 12 23:01:00 2012 +0200
Added comments in the example yaml file as requested
commit 7f9718f185ec991bc165c4a52b2468cf41f4c349
Author: Dag Wieërs <dag@wieers.com>
Date: Thu Jul 12 22:49:38 2012 +0200
Add the default nose color too, to test specific overrides
commit eb63b9e899318ce0c26902ca73af50135a6224e4
Author: Dag Wieërs <dag@wieers.com>
Date: Thu Jul 12 22:44:35 2012 +0200
Introduce comics and cartoons to test yaml groups defined on a per-node basis
commit aa13d233078b825a8057bebf35ed478342cf4e43
Author: Dag Wieërs <dag@wieers.com>
Date: Thu Jul 12 19:33:15 2012 +0200
A small fix to revert to old state
commit 264ebaa77c4243f2e9117e8d1168dc2f2eed7ee2
Author: Dag Wieërs <dag@wieers.com>
Date: Thu Jul 12 19:31:51 2012 +0200
Combine both yaml unit tests into one example file
commit 7db49a8048e78402c4c9a0a6cb2604689280fbbb
Author: Dag Wieërs <dag@wieers.com>
Date: Thu Jul 12 16:46:53 2012 +0200
Might as well fix this too
commit f36c6c8c5b419865939c7e2b0b26f6c97255fdc8
Author: Dag Wieërs <dag@wieers.com>
Date: Thu Jul 12 16:42:00 2012 +0200
Added unit tests for host-groups patch
For the unit test I chose to keep the original yaml file in place as a reference.
This patch also includes a fix.
commit a96f6813522c5ae8b2be4514a2de56a775c6b7b0
Author: Dag Wieërs <dag@wieers.com>
Date: Thu Jul 12 12:30:43 2012 +0200
Allow groups to be defined on a per-host basis
This makes it possible to define on a per-host basis what groups a host is in.
When managing a large set of systems it makes it easier to ensure each of the
systems is defined in a set of groups (e.g. production/qa/development,
linux/solaris/aix) rather than having to add systems to multiple disconnected
groups.
----
- host: system01
- host: system02
- host: system03
- group: linux
hosts:
- system01
- system02
- group: solaris
hosts:
- system03
- group: production
hosts:
- system01
- system03
- group: qa
- system02
- group: dbserver
hosts:
- system01
- group: ntpserver
hosts:
- system02
- group: webserver
- system03
----
Can be redefined as:
----
- host: system01
groups: [ linux, production, dbserver ]
- host: system02
groups: [ linux, qa, ntpserver ]
- host: system03
groups: [ solaris, production, webserver ]
----
get_hosts() was treating [] (meaning complete restriction, no hosts allowed)
the same as None (meaning no restriction, all hosts allowed). Fixed logic.
