John R Barker
72bfb051c3
Allow ansible.cfg overrides ( #27372 )
...
If a cfg file exists for this this command then use it.
This allows versioned test configuration to override the built in
Ansible constants.
2017-07-27 11:37:15 +01:00
Toshio Kuratomi
9f7b0dfc30
Remove automatic use of system six
...
* Enable the pylint no-name-in-module check. Checks that identifiers in
imports actually exist. When we do this, we also have to ignore
_MovedItems used in our bundled six. This means pylint won't check
for bad imports below ansible.module_utils.six.moves but that's
something that pylint punts on with a system copy of six so this is
still an improvement.
* Remove automatic use of system six. The exec in the six code which
tried to use a system library if available destroyed pylint's ability
to check for imports of identifiers which did not exist (the
no-name-in-module check). That test is important enough that we
should sacrifice the bundling detection in favour of the test.
Distributions that want to unbundle six can replace the bundled six in
ansible/module_utils/six/__init__.py to unbundle. however, be aware
that six is tricky to unbundle. They may want to base their efforts
off the code we were using:
2fff690caa/lib/ansible/module_utils/six/__init__.py
* Update tests for new location of bundled six Several code-smell tests
whitelist the bundled six library. Update the path to the library so
that they work.
* Also check for basestring in modules as the enabled pylint tests will
also point out basestring usage for us.
2017-07-25 15:58:23 -07:00
Matt Clay
27033915cc
Add --list-targets and speed up --explain. ( #26838 )
...
* Add ansible-test --list-targets option.
* Speed up ansible-test integration --explain.
2017-07-14 19:11:25 -07:00
Matt Clay
e63f69464f
Add ansible-test options and docs. ( #26823 )
...
* All integration commands support --continue-on-error
* The network-integration command supports --inventory
* Add landing page for compile test docs.
* Add bot documentation links.
2017-07-14 16:52:11 -07:00
Matt Clay
7931e11437
Use subset of Windows tests for "all" tests. ( #26830 )
2017-07-14 16:51:32 -07:00
Matt Clay
789218c215
Initial ansible-test sanity docs. ( #26775 )
...
* Rename no-iterkeys test for consistency.
* Require docs for all ansible-test sanity tests.
* Initial ansible-test sanity docs.
* Fix capitalization of Python.
* Fix sanity code smell test false positives.
* Fix another code-smell false positive.
2017-07-14 14:24:45 +01:00
Pilou
4b3d6dfa8a
Use pycodestyle instead of pep8 ( #25947 )
2017-07-13 11:46:31 -07:00
Matt Clay
fad469efca
Ignore platform_agnostic.yaml in ansible-test.
2017-07-13 11:23:59 -07:00
Matt Clay
0b784c65b1
Add sanity import test to ansible-test. ( #26730 )
...
* Add sanity import test to ansible-test.
* Run sanity import test on all python versions.
2017-07-12 20:04:22 -07:00
Matt Clay
20d49af475
Add ANSIBLE_TEST_CI env var to integration tests.
2017-07-12 17:46:12 -07:00
Matt Clay
ab5dd24430
Handle pip/python errors in ansible-test better.
2017-07-11 16:37:16 -07:00
Matt Clay
82558baaf6
Add role dependencies to change classification.
2017-07-10 18:20:09 -07:00
Matt Clay
4a013d6d69
Add RHEL 7.3 to ansible-test and CI. ( #26550 )
...
* Add support for RHEL to ansible-test.
* Enable RHEL 7.3 on Shippable.
2017-07-07 16:52:40 -07:00
David Newswanger
294c0f91c0
added option to ansible-test to run test playbooks using the debug strategy ( #26499 )
2017-07-07 12:37:08 -07:00
Matt Clay
a94156227d
Improve error handling for cs tests.
2017-07-06 17:55:51 -07:00
Matt Clay
45e377566c
Refactor ansible-test config classes. ( #26505 )
...
* Move Config classes from executor.py to config.py.
* Move Environment and Test config to config.py.
* Move Coverage/CoverageReport Config to config.py.
* Clean up type hints.
2017-07-06 16:14:44 -07:00
Matt Clay
bd8ea89b1b
Improve handling of network test changes.
2017-07-06 14:25:58 -07:00
Matt Clay
c26a556929
Add check for known_hosts changing during tests. ( #26470 )
...
* Add check for known_hosts changing during tests.
* Don't use known hosts file for delegate_to test.
2017-07-06 00:47:28 -07:00
Matt Clay
fba76444e0
Add ansible-test coverage report --show-missing
.
2017-07-05 17:11:41 -07:00
David Newswanger
0a6dcf4169
changed ansible-test so that it replaces . with - instead of . with _ so that the hostname is valid
2017-07-03 20:11:50 -07:00
Matt Clay
cd3a795e65
Permissions fix for remote coverage downloads.
2017-07-02 14:09:16 -07:00
Matt Clay
554423cf04
Expose docker socket to test containers.
2017-06-30 16:46:41 -07:00
René Moser
77d0542985
test, integration: add diff support ( #26296 )
2017-06-30 20:43:34 +02:00
Pierre-Louis Bonicoli
d08ff86169
ansible-test: pass --[skip-]tags to ansible-playbook
...
Thanks to Olaf Klischat for the idea.
2017-06-30 10:35:39 -07:00
Matt Clay
e56ea435f0
Skip dependency check for deleted files. ( #26112 )
2017-06-26 11:48:46 -07:00
David Newswanger
b2d88f8b73
updated ansible-test integration to skip any files in test/integration/targets/ that aren't directories.
...
Fixes #26007
2017-06-22 13:03:42 -07:00
jctanner
996dcb989c
Add the cloud provider script for the vcenter simulator container ( #25964 )
2017-06-22 10:49:58 -07:00
Matt Clay
53180ebb57
Fix Shippable BASE_BRANCH lookup on merges.
2017-05-19 01:51:24 +08:00
Matt Clay
5babe2daea
Increase python version coverage for tests. ( #24762 )
...
* Improve ansible-test inventory handling.
* Fix python 3 re-raise of exception from thread.
* Fix python 3 encoding for windows-integration.
* Run network tests on multiple python versions.
* Run windows tests on multiple python versions.
* Support Shippable delegation using --tox.
* Skip vyos_command on python 3 tests until fixed.
* Add python 3 filtering to local and tox.
* Fix tests to support back to back runs.
* Temporarily test networking with python 2.7 only.
Running the tests back to back causes intermittent test failures
which need to be addressed before we can test multiple versions
in a single test run.
2017-05-19 01:37:53 +08:00
Matt Clay
4daa16f9ce
Add --all and --stub coverage options.
...
ci_coverage
ci_complete
2017-05-16 12:02:03 +08:00
Matt Clay
6a2a7a2392
Enable codecov.io and add coverage grouping.
2017-05-15 19:14:05 +08:00
Matt Clay
fdcad0f44d
Improve on-demand coverage for Shippable.
2017-05-14 18:27:58 +08:00
John R Barker
e9e2d67035
Run vyos_command Network tests in Shippable ( #24514 )
...
* Run VyOS Network tests in Shippable
* net_command no longer exists
* network/ci
* Disable other network tests
* pep8 fix
* Fix PEP 8 issue.
2017-05-12 12:49:12 +01:00
Matt Clay
dfd19a812f
Miscellaneous bug fixes for ansible-test.
...
- Overhauled coverage injector to fix issues with non-local tests.
- Updated integration tests to work with the new coverage injector.
- Fix concurrency issue by using random temp files for delegation.
- Fix handling of coverage files from root user.
- Fix handling of coverage files without arcs.
- Make sure temp copy of injector is world readable and executable.
2017-05-12 14:55:48 +08:00
Matt Clay
fb9efd3c7e
Add --remote-terminate
option to ansible-test.
...
Set `--remote-terminate success` for osx tests.
2017-05-11 22:42:52 +08:00
Matt Clay
b8cb3f519b
Detect and fix environment tampering in tests.
2017-05-11 19:45:15 +08:00
Matt Clay
e6934e42be
Update ansible-test change handling and CS plugin. ( #24395 )
...
* Use CloudStack simulator from Ansible repo.
* Recognize cloud specific ansible-test changes.
* Hide ansible-test cs notice in explain mode.
2017-05-09 16:15:27 +08:00
Matt Clay
0573b8bcd2
Fix ansible-test cloud integration bugs.
2017-05-05 21:46:29 +08:00
Matt Clay
ac72fd9d2c
Add missing return statements.
2017-05-05 18:37:11 +08:00
Matt Clay
201242f302
Fix call to wrong method in ansible-test.
2017-05-05 18:18:40 +08:00
Matt Clay
1582e9bf7d
Fix handling of Shippable env vars.
2017-05-05 17:44:51 +08:00
Matt Clay
a07d42e16d
Add support for cloud tests to ansible-test. ( #24315 )
...
* Split out ansible-test docker functions.
* Add cloud support to ansible-test.
2017-05-05 16:23:00 +08:00
Matt Clay
6522d703a1
Fix ansible-test pylint and pep8 issues. ( #24245 )
...
* Fix pylint useless-super-delegation issues.
* Fix miscellaneous pylint issues.
* Fix PEP 8 issues.
* Fix warnings reported by PyCharm.
2017-05-03 23:19:44 +08:00
Matt Clay
d662f6f0db
Fix bugs in ansible-test units command. ( #24044 )
...
* Handle old versions of coverage.
* Handle old versions of setuptools.
* Detect python version for docker/remote units.
* Add sanity override for test constraints.
2017-04-27 11:21:11 +08:00
Matt Clay
cb1f57d4e5
Add rstcheck to ansible-test and correct issues. ( #23550 )
...
* Add rstcheck to ansible-test.
* Fix rst code-block languages and syntax errors.
* Fix rst inline literals.
* Update python 2 code block to pass tests on py 3.
2017-04-13 10:28:52 -07:00
Matt Clay
92ad9a19b6
Fix ansible-test empty diff handling on Shippable.
2017-04-08 22:08:08 -07:00
Matt Clay
0cf1e8e061
Fix ansible-test handling of import/syntax errors.
2017-04-07 12:20:31 -07:00
Matt Clay
1daa69d685
Initial pylint support for ansible-test sanity.
2017-04-04 16:16:21 -07:00
Matt Clay
4fdeade389
Overhaul ansible-test import analysis. ( #22888 )
...
- Add support for module_utils packages for better organization.
- Add support for "virtual" module_utils packages such as `six`.
2017-03-22 18:07:53 -07:00
Matt Clay
042e387f27
Fix metadata handling for shell command. ( #22801 )
2017-03-20 15:31:57 -07:00
Matt Clay
991bf0a1c3
Fix ansible-test base branch handling on merge. ( #22759 )
2017-03-17 23:33:32 -07:00
Matt Clay
47887a2edd
Fix ansible-test base branch usage on Shippable.
2017-03-17 17:35:09 -07:00
Matt Clay
f7c83ccdc0
Fix ansible-test PR diff handling.
...
Permits use of PRs on branches other than devel.
2017-03-16 10:34:31 -07:00
Matt Clay
3a22467b38
Remove python 2.4 compile test.
2017-03-15 15:46:12 -07:00
Matt Clay
869449e288
Add test verification to ansible-test. ( #22636 )
...
* Add unified git diff parser.
* Add metadata and diff handling.
* Add test confidence/verification to bot output.
2017-03-15 12:17:42 -07:00
Matt Clay
5e9a2b8528
Verify SHA in Shippable change detection.
2017-03-15 11:31:55 -07:00
Matt Clay
89559f78de
Add --failure-ok
option to ansible-test
. ( #22623 )
2017-03-14 16:32:21 -07:00
Matt Clay
771b14acb2
Sort test messages for deterministic output. ( #22581 )
2017-03-13 20:45:10 -07:00
Matt Clay
cfd313bced
Disable ansible color output on sanity tests.
2017-03-08 12:18:16 -08:00
Matt Clay
3ab344e8bd
Use lint/junit output for compile test. ( #22395 )
...
* Refactor sanity classes for use in all tests.
* Use lint/junit output for compile test.
* Add missing options for compile test.
* Fix early bailout on requirements install.
2017-03-08 00:47:21 -08:00
Matt Clay
fbf55886ba
Use lint friendly output for python syntax checks. ( #22390 )
2017-03-07 20:27:50 -08:00
Matt Clay
8f463fcdd2
Create bot friendly sanity output. ( #22381 )
2017-03-07 14:59:50 -08:00
Matt Clay
3e4be156d7
Pre-compile regexes to speed up target processing.
...
Without this, changing a large number of files results in target
processing taking a very long time due to repeatedly compiling
the same patterns in a loop over many targets.
2017-03-06 12:35:12 -08:00
Matt Clay
d66ce40ecb
Overhaul ansible-test sanity
implementation. ( #22177 )
...
- Tests are run to completion instead of stopping on first failure.
- Test results are now parsed instead of passing through to the console.
- Test results can be saved in junit xml format.
- Test results will show up on the Shippable "Tests" result tab.
- Added an experimental --lint option for easier integration with other tools.
- Code smell tests are now usable with the --list-tests, --test and --skip-test options.
- Code split out from executor.py into sanity.py.
- Rename download-logs to download.py and add support for test and coverage results.
- Miscellaneous improvements.
2017-03-02 12:36:46 -08:00
Matt Clay
28b882e737
Support ansible commands in sanity tests. ( #22172 )
2017-03-01 15:48:14 -08:00
Matt Clay
4f3dade266
Add retries for ansible-test docker run.
2017-02-27 22:55:46 -08:00
Matt Clay
45cec93e99
Disable newer shellcheck tests which don't pass.
2017-02-27 22:31:47 -08:00
Matt Clay
fcac261eef
Run unit tests in isolation w/ coverage support.
2017-02-27 17:16:45 -08:00
Matt Clay
c87ebb6215
Deduplicate arcs when combining coverage data. ( #22022 )
2017-02-27 15:46:15 -08:00
Matt Clay
2797cff773
Run destructive win tests when creating instances.
2017-02-27 10:36:53 -08:00
Matt Clay
2804d2e848
Split up unit tests on Shippable. ( #21931 )
...
* Add more verbose output to coverage combine.
* Run unit tests on CI separately by version.
2017-02-24 16:58:56 -08:00
Matt Clay
bc8186543a
Pass base branch to module validator on delegate.
2017-02-20 14:28:11 -08:00
Matt Martz
9b1bd2eb7e
Re-enable module comparisons in validate-modules ( #21582 )
...
* Re-enable module comparisons, specifically for new module detection and for finding new options/arguments
* Only do new module checks in shippable, local will display warning
2017-02-17 15:13:18 -08:00
Matt Clay
1ce2ea0b6c
Fix ansible-test handling of deleted test targets.
2017-02-17 11:16:24 -08:00
Matt Clay
2192c1eb02
Fix ansible-test selection of inventory file.
2017-02-16 15:03:27 -08:00
John R Barker
5e28d3a15c
network_cli shouldn't be specified anymore ( #21458 )
2017-02-15 12:58:12 +00:00
Matt Clay
d54bc09fae
Consider module_utils deps when running tests. ( #21382 )
...
* Skip pep8 analysis when --explain is used.
* Fix return type annotations.
* Match line length requirement of PEP 8 config.
* Consider module_utils deps when running tests.
2017-02-13 18:49:36 -08:00
Peter Sprygada
7f1c43e597
updates ios modules to support persistent socket ( #21258 )
...
* updates all ios modules to support persistent socket
* adds ios action plugin to connect to device
* adds exec_command() to ios shared module
* fixes ios_config and ios_template local action
* update all unit test cases
* adds base test module for ios module testing
2017-02-13 15:22:14 +00:00
Matt Clay
4189592701
Revert "Enable process isolation for unit tests."
...
This reverts commit 91526cd9f2
.
Removing this feature primarily because it interferes with
collecting proper code coverage results. I may restore the
feature later if that can be resolved.
2017-02-09 11:10:06 -08:00
Matt Clay
91526cd9f2
Enable process isolation for unit tests.
2017-02-08 14:36:42 -08:00
Matt Clay
bce5fc7d96
Use us-east-2 for Windows CI on Shippable.
2017-02-06 15:01:20 -08:00
Matt Clay
309a37de86
Add --debug
option to ansible-test
.
2017-02-01 16:05:03 -08:00
Matt Clay
6ef1a6aeb6
Add pep8
to ansible-test
. ( #20745 )
...
The rule sets and legacy file list are a first draft.
It is likely that they will need to be revised.
2017-01-27 12:34:00 -08:00
Dag Wieers
3a0a74dc18
Make test/runner less verbose when spawning instances
...
As discussed on IRC
2017-01-27 12:27:46 -08:00
Matt Clay
5503b89831
Run fewer tests for winrm config script changes.
2017-01-26 15:54:01 -08:00
Matt Clay
76d563af79
Temporarily switch Windows tests to us-east-1.
...
Will switch back to us-east-2 after AWS limit increase is done.
2017-01-26 07:37:14 -08:00
Matt Clay
fa7f84217b
Add startup checks to ansible-test.
2017-01-24 13:08:04 -08:00
Matt Clay
61e6e7868c
Improve jinja2 test coverage. ( #20533 )
...
* Run `template` test on latest jinja2 version.
* Update jinja2 install for groupby_filter.
* Clean test output dir for each test.
2017-01-20 16:38:52 -08:00
Matt Clay
7ed67746cf
Enable Windows 2016 on Shippable.
...
Also launch Windows instances on us-east-2 and all others on us-east-1.
2017-01-18 18:37:42 -08:00
Matt Clay
4ba9f63afe
Use multiple AWS regions in ansible-test. ( #20429 )
...
* Move ansible-test EnvironmentConfig to util.
* Use multiple AWS regions in ansible-test.
2017-01-18 16:31:34 -08:00
Matt Clay
4673a9c9c2
Generate ssh key once for network tests.
2017-01-18 13:33:12 -08:00
Matt Clay
4c97bad28e
Update ansible-test network change classification.
2017-01-18 12:19:31 -08:00
Matt Clay
3c51ce56fd
Enable more network support in ansible-test. ( #20372 )
...
* Only start platform instances with tests selected.
* Enable ios on Shippable.
* Show inventory in explain mode.
* Fix indentation of generated network inventory.
* Update classification of network module_utils.
2017-01-17 16:24:05 -08:00
Matt Clay
dbb89549ff
Improve ansible-test error handling and timeouts. ( #20328 )
...
* Eliminate warning/retry on old instance check.
* Increase instance start timeout for windows.
2017-01-16 18:59:44 -08:00
Matt Clay
b5d5888b6e
Add support for ios to ansible-test.
2017-01-16 15:05:56 -08:00
Matt Clay
e2bd71ac02
Clean up inventory generation in ansible-test. ( #20313 )
2017-01-16 12:31:17 -08:00
Matt Clay
a59ce3a7f9
Add retry on failure to get instance.
2017-01-13 18:35:28 -08:00
Matt Clay
f3130898bb
Retry instance start on error.
2017-01-13 16:49:30 -08:00
Matt Clay
d8733a5455
Complete initial network-integration support.
2017-01-12 12:39:00 -08:00
Matt Clay
300181cfd3
Send current winrm config to ansible-core-ci.
2017-01-10 20:05:47 -08:00
Matt Clay
f340b8d14f
Add test support for python 3.6.
2017-01-10 14:34:42 -08:00
Matt Clay
3ec7243918
Run all win tests for winrm config script changes.
2017-01-08 00:57:51 -08:00
Brian Coca
57f8b791d6
consolidated docs
...
point to new doc locations
removed non existing dirs
2017-01-06 09:16:59 -05:00
Matt Clay
b7780ca530
Pass DOCKER_* env vars to docker in ansible-test.
2017-01-04 13:23:16 -08:00
Matt Clay
dac226de4a
Improve ansible-test completion and messaging. ( #19627 )
...
* Simplify type annotation of args.docker_util.
* Improve error messaging for ansible-doc test.
* Add completion for docker, remote and windows.
2016-12-22 09:57:17 -08:00
Matt Clay
422857166b
Remove legacy test alias migration script. ( #19629 )
...
* Require code-smell tests to be files.
* Add sanity check for integration test aliases.
* Remove migration script for test aliases.
2016-12-21 16:46:28 -08:00
Matt Clay
9b5c782a0b
Use docker pull
by default in ansible-test.
2016-12-14 20:11:43 -08:00
Matt Clay
f28b5a0ed8
Add --tox-sitepackages option to ansible-test.
2016-12-13 16:57:23 -08:00
Matt Clay
45772b961f
Add private fork CI support to ansible-test.
2016-12-13 15:49:57 -08:00
Matt Clay
1c9a58a885
Run all tests when no last successful commit.
...
(cherry picked from commit 2e56035fddfd04eb084bd977fc1ee2f35c2f5283)
2016-12-08 11:35:20 -05:00
Matt Clay
5628e260f2
Relocate module unit tests. ( #18812 )
...
* Relocate module unit tests.
* Fix classification of unit test changes.
2016-12-08 11:35:20 -05:00
Matt Clay
573078fc6b
Fix unit test directory path target.
2016-12-08 11:35:20 -05:00
Matt Clay
47083f145a
Don't error if pytest skips all unit tests.
2016-12-08 11:35:20 -05:00
Matt Clay
8e951b9f98
Remove legacy test/samples
directory.
2016-12-08 11:35:20 -05:00
Matt Clay
7d3e2b2121
Verify path before assigning to sanity test.
2016-12-08 11:35:19 -05:00
Matt Clay
14a2757116
Correct change classification for network plugins.
2016-12-05 20:07:18 -05:00
Adrian Likins
ba56c87566
Add SSH_AUTH_SOCK to default env
...
So ssh-agent works when using locally.
2016-11-30 15:53:15 -08:00
Matt Clay
bee58fe890
Fix compile skip path handling for targeted tests.
2016-11-30 09:47:56 -08:00
Matt Clay
6bbd92e422
Initial ansible-test implementation. ( #18556 )
2016-11-29 21:21:53 -08:00