A job exists that runs hammer-tests every 4 days. This runs against nightly production repositories and has been red for many months. The hammer-tests repository has not seen an update since Oct, 2017 and has a small test suite. I’d like to get input on what to do with both the job and the repository:
This is the first I’ve heard of hammer-tests, so I don’t think I have much say here. Seems like it might duplicate some integration testing covered by bats?
I believe there were some benefits, cc @Tomas_Strachota (framework) and @mbacovsky (tax tests) explicitly.
IIRC major tests were around taxonomies for which we didn’t have alternative. Could you guys check if they are red because of real regressions? I think if they are red for a long time, we either need to find a maintainer who will be notified or we can drop them.
The purpose of this tests was to test hammer in complete integration scenarios - similar to what bats tests do. Compared to bats, hammer tests provide some additional features for more convenient debugging of failures. They collect hammer output, logs and server logs for each executed command so it’s faster and easier to see causes of failures.
My initial idea (in 2014!) was to add tests with each PR that I’m testing. As a reviewer I have to type the commands into console anyway so it doesn’t make any difference if I type them into a file instead. Nice benefit of this approach was that I could easily re-run my steps when a PR was updated. The problematic part here was someone then had to review and merge the tests. The review usually took long and also I didn’t want to spend more time iterating on improving a PR with tests. Even @mbacovsky’s tests for taxonomies that @Marek_Hulan mentioned are still only in a branch somewhere and never made it to a PR.
Even though I see value in the approach I described above, adoption of the tests is minimal. I advertised them several times in past but it didn’t have much effect. The fact they have been red for several months and nobody cares says something. I would say if we agree there’s some value in the tests, we have to find a maintainer, fix them and start contributing more frequently. If not, let’s just throw them away.
Would you say their purpose was to validate hammer changes or to validate that server changes don’t break hammer?
Both is true to some extent. I’d define the purpose as “to validate that hammer is functional in time regardless of where the changes were made”.
Here is output of current failures, if we want to fix them. I think if we do want to fix them, that we should run these by default whenever we run bats tests to incorporate them and drop the unmonitored test.
Thus, the question is still do we fix these or drop them.
fatal: [pipeline-foreman-nightly-centos7]: FAILED! => changed=true
cmd: ./run_tests ./tests/
delta: '0:01:16.264086'
end: '2018-09-27 14:37:18.100155'
msg: non-zero return code
rc: 1
start: '2018-09-27 14:36:01.836069'
stderr: ''
stderr_lines: []
stdout: |-
organization
create
hammer organization create --name org33[0;36;49m [command #1][0m
[0;34;49mCould not create the organization:
The server does not support such operation.[0m
[0;31;49m[FAIL] [0mreturns ok
user
create
hammer --csv user create --login some_user33 --mail some.use ...[0;36;49m [command #2][0m
[0;32;49m[ OK ] [0mreturns ok
assign to organization
hammer organization add-user --name org33 --user some_user33[0;36;49m [command #3][0m
[0;34;49mCould not associate the user:
404 Not Found[0m
[0;31;49m[FAIL] [0mreturns ok
hammer organization remove-user --name org33 --user some_user33[0;36;49m [command #4][0m
[0;34;49mCould not disassociate the user:
404 Not Found[0m
[0;31;49m[FAIL] [0mreturns ok
info
hammer user info --login some_user33[0;36;49m [command #5][0m
[0;32;49m[ OK ] [0mreturns ok
[0;32;49m[ OK ] [0mhas column Id
[0;32;49m[ OK ] [0mhas column Login
[0;32;49m[ OK ] [0mhas column Name
[0;32;49m[ OK ] [0mhas column Email
architecture
list
hammer architecture list[0;36;49m [command #6][0m
[0;32;49m[ OK ] [0mreturns ok
[0;32;49m[ OK ] [0mhas column Id
[0;32;49m[ OK ] [0mhas column Name
create
hammer architecture create --name arch33[0;36;49m [command #7][0m
[0;32;49m[ OK ] [0mreturns ok
operating system
create
hammer --csv os create --name test_os33 --major 6 --minor 3 ...[0;36;49m [command #8][0m
[0;32;49m[ OK ] [0mreturns ok
partition table
create
hammer partition-table create --name ptable33 --file /root/h ...[0;36;49m [command #9][0m
[0;32;49m[ OK ] [0mreturns ok
dump
hammer partition-table dump --name ptable33[0;36;49m [command #10][0m
[0;32;49m[ OK ] [0mreturns ok
[0;32;49m[ OK ] [0mdumps the content
add/remove os
hammer partition-table add-operatingsystem --operatingsystem ...[0;36;49m [command #11][0m
[0;32;49m[ OK ] [0mreturns ok
os associated
hammer partition-table info --name ptable33[0;36;49m [command #12][0m
[0;32;49m[ OK ] [0mreturns ok
[0;32;49m[ OK ] [0mOperating systems value
hammer partition-table remove-operatingsystem --operatingsys ...[0;36;49m [command #13][0m
[0;32;49m[ OK ] [0mreturns ok
installation medium
create
hammer --csv medium create --name medium33 --path http://mir ...[0;36;49m [command #14][0m
[0;32;49m[ OK ] [0mreturns ok
assign to organization
hammer organization add-medium --name org33 --medium medium33[0;36;49m [command #15][0m
[0;34;49mCould not associate the medium:
404 Not Found[0m
[0;31;49m[FAIL] [0mreturns ok
hammer organization remove-medium --name org33 --medium medium33[0;36;49m [command #16][0m
[0;34;49mCould not disassociate the medium:
404 Not Found[0m
[0;31;49m[FAIL] [0mreturns ok
add/remove os
hammer medium add-operatingsystem --operatingsystem test_os3 ...[0;36;49m [command #17][0m
[0;32;49m[ OK ] [0mreturns ok
os associated
hammer medium info --name medium33[0;36;49m [command #18][0m
[0;32;49m[ OK ] [0mreturns ok
[0;32;49m[ OK ] [0mOperating systems value
hammer medium remove-operatingsystem --operatingsystem test_ ...[0;36;49m [command #19][0m
[0;32;49m[ OK ] [0mreturns ok
template
create
hammer --csv template create --name tpl33 --file /root/hamme ...[0;36;49m [command #20][0m
[0;32;49m[ OK ] [0mreturns ok
dump
hammer template dump --name tpl33[0;36;49m [command #21][0m
[0;32;49m[ OK ] [0mreturns ok
[0;32;49m[ OK ] [0mdumps the content
assign to organization
hammer organization add-config-template --name org33 --confi ...[0;36;49m [command #22][0m
[0;34;49mCould not associate the configuration template:
404 Not Found[0m
[0;31;49m[FAIL] [0mreturns ok
hammer organization remove-config-template --name org33 --co ...[0;36;49m [command #23][0m
[0;34;49mCould not disassociate the configuration template:
404 Not Found[0m
[0;31;49m[FAIL] [0mreturns ok
add/remove os
hammer template add-operatingsystem --operatingsystem test_o ...[0;36;49m [command #24][0m
[0;32;49m[ OK ] [0mreturns ok
os associated
hammer template info --name tpl33[0;36;49m [command #25][0m
[0;32;49m[ OK ] [0mreturns ok
[0;32;49m[ OK ] [0mOperating systems value
hammer template remove-operatingsystem --operatingsystem tes ...[0;36;49m [command #26][0m
[0;32;49m[ OK ] [0mreturns ok
kinds
hammer template kinds[0;36;49m [command #27][0m
[0;32;49m[ OK ] [0mreturns ok
hardware model
create
hammer model create --name model33 --info some model info -- ...[0;36;49m [command #28][0m
[0;32;49m[ OK ] [0mreturns ok
info
hammer model info --name model33[0;36;49m [command #29][0m
[0;32;49m[ OK ] [0mreturns ok
[0;32;49m[ OK ] [0mhas column Id
[0;32;49m[ OK ] [0mhas column Name
[0;32;49m[ OK ] [0mhas column Vendor class
[0;32;49m[ OK ] [0mhas column HW model
[0;32;49m[ OK ] [0mhas column Info
[0;32;49m[ OK ] [0mName value
[0;32;49m[ OK ] [0mHW model value
[0;32;49m[ OK ] [0mVendor class value
operating system
add architecture
hammer os add-architecture --id 2 --architecture arch33[0;36;49m [command #30][0m
[0;32;49m[ OK ] [0mreturns ok
add partition table
hammer os add-ptable --id 2 --partition-table ptable33[0;36;49m [command #31][0m
[0;32;49m[ OK ] [0mreturns ok
add template
hammer os add-config-template --id 2 --config-template tpl33[0;36;49m [command #32][0m
[0;32;49m[ OK ] [0mreturns ok
add medium
hammer os update --id 2 --medium-ids 10[0;36;49m [command #33][0m
[0;32;49m[ OK ] [0mreturns ok
info by id
hammer os info --id 2[0;36;49m [command #34][0m
[0;32;49m[ OK ] [0mreturns ok
[0;32;49m[ OK ] [0mhas column Id
[0;32;49m[ OK ] [0mhas column Name
[0;32;49m[ OK ] [0mhas column Release name
[0;32;49m[ OK ] [0mhas column Family
[0;32;49m[ OK ] [0mhas column Installation media
[0;32;49m[ OK ] [0mhas column Architectures
[0;32;49m[ OK ] [0mhas column Partition tables
[0;32;49m[ OK ] [0mhas column Templates
[0;32;49m[ OK ] [0mhas column Parameters
[0;32;49m[ OK ] [0mId value
[0;32;49m[ OK ] [0mTitle value
[0;32;49m[ OK ] [0mName value
[0;32;49m[ OK ] [0mRelease name value
[0;32;49m[ OK ] [0mFamily value
[0;32;49m[ OK ] [0mInstallation media value
[0;32;49m[ OK ] [0mArchitectures value
[0;32;49m[ OK ] [0mPartition tables value
[0;32;49m[ OK ] [0mTemplates value
architecture
info
hammer architecture info --name arch33[0;36;49m [command #35][0m
[0;32;49m[ OK ] [0mreturns ok
[0;32;49m[ OK ] [0mhas column Id
[0;32;49m[ OK ] [0mhas column Name
[0;32;49m[ OK ] [0mhas column Operating systems
[0;32;49m[ OK ] [0mName value
[0;32;49m[ OK ] [0mOperating systems value
installation medium
info
hammer medium info --name medium33[0;36;49m [command #36][0m
[0;32;49m[ OK ] [0mreturns ok
[0;32;49m[ OK ] [0mhas column Id
[0;32;49m[ OK ] [0mhas column Path
[0;32;49m[ OK ] [0mhas column OS Family
[0;32;49m[ OK ] [0mhas column Operating systems
[0;32;49m[ OK ] [0mName value
[0;32;49m[ OK ] [0mOS Family value
[0;32;49m[ OK ] [0mOperating systems value
[0;32;49m[ OK ] [0mPath value
partition table
info
hammer partition-table info --name ptable33[0;36;49m [command #37][0m
[0;32;49m[ OK ] [0mreturns ok
[0;32;49m[ OK ] [0mhas column Id
[0;32;49m[ OK ] [0mhas column Name
[0;32;49m[ OK ] [0mhas column OS Family
[0;32;49m[ OK ] [0mName value
[0;32;49m[ OK ] [0mOS Family value
[0;32;49m[ OK ] [0mOperating systems value
template
info
hammer template info --name tpl33[0;36;49m [command #38][0m
[0;32;49m[ OK ] [0mreturns ok
[0;32;49m[ OK ] [0mhas column Id
[0;32;49m[ OK ] [0mhas column Name
[0;32;49m[ OK ] [0mhas column Type
[0;32;49m[ OK ] [0mhas column Operating systems
[0;32;49m[ OK ] [0mName value
[0;32;49m[ OK ] [0mType value
[0;32;49m[ OK ] [0mOperating systems value
smart proxy
find or create create
hammer --csv proxy list --search url=https://pipeline-forema ...[0;36;49m [command #39][0m
info
hammer proxy info --name pipeline-foreman-nightly-centos7.ex ...[0;36;49m [command #40][0m
[0;32;49m[ OK ] [0mreturns ok
[0;32;49m[ OK ] [0mhas column Id
[0;32;49m[ OK ] [0mhas column Name
[0;32;49m[ OK ] [0mhas column URL
[0;32;49m[ OK ] [0mhas column Features
[0;32;49m[ OK ] [0mId value
[0;32;49m[ OK ] [0mName value
[0;32;49m[ OK ] [0mURL value
import classes
hammer proxy import-classes --name pipeline-foreman-nightly- ...[0;36;49m [command #41][0m
[0;32;49m[ OK ] [0mreturns ok
deletion
domain
create
hammer domain create --name domain33 --description domain de ...[0;36;49m [command #42][0m
[0;32;49m[ OK ] [0mreturns ok
parameters
hammer domain set-parameter --domain domain33 --name param_a ...[0;36;49m [command #43][0m
[0;32;49m[ OK ] [0mreturns ok
hammer domain set-parameter --domain domain33 --name param_b ...[0;36;49m [command #44][0m
[0;32;49m[ OK ] [0mreturns ok
hammer domain delete-parameter --domain domain33 --name param_b[0;36;49m [command #45][0m
[0;32;49m[ OK ] [0mreturns ok
info
hammer domain info --name domain33[0;36;49m [command #46][0m
[0;32;49m[ OK ] [0mreturns ok
[0;32;49m[ OK ] [0mhas column Id
[0;32;49m[ OK ] [0mhas column Name
[0;32;49m[ OK ] [0mhas column Description
[0;32;49m[ OK ] [0mhas column DNS Id
[0;32;49m[ OK ] [0mhas column Parameters
[0;32;49m[ OK ] [0mName value
[0;32;49m[ OK ] [0mDescription value
deletion
hammer domain delete --name domain33[0;36;49m [command #47][0m
[0;32;49m[ OK ] [0mreturns ok
role
create
hammer role create --name role33[0;36;49m [command #48][0m
[0;32;49m[ OK ] [0mreturns ok
list
hammer role list[0;36;49m [command #49][0m
[0;32;49m[ OK ] [0mcontains created role
filters
hammer role filters --name role33[0;36;49m [command #50][0m
[0;32;49m[ OK ] [0mreturns ok
update
hammer role update --new-name role33_2 --name role33[0;36;49m [command #51][0m
[0;32;49m[ OK ] [0mreturns ok
filter
available permissions and resources
permissions
hammer filter available-permissions[0;36;49m [command #52][0m
[0;32;49m[ OK ] [0mreturns ok
resources
hammer filter available-resources[0;36;49m [command #53][0m
[0;32;49m[ OK ] [0mreturns ok
create
hammer --csv filter create --role role33_2 --permission-ids ...[0;36;49m [command #54][0m
info
hammer filter info --id 159[0;36;49m [command #55][0m
[0;32;49m[ OK ] [0mreturns ok
list
hammer filter list[0;36;49m [command #56][0m
[0;32;49m[ OK ] [0mreturns ok
update
hammer filter update --id 159 --permission-ids 3,4 --search ...[0;36;49m [command #57][0m
[0;32;49m[ OK ] [0mreturns ok
deletions
filter
hammer filter delete --id 159[0;36;49m [command #58][0m
[0;32;49m[ OK ] [0mreturns ok
role
hammer role delete --name role33_2[0;36;49m [command #59][0m
[0;32;49m[ OK ] [0mreturns ok
deletions
organization
hammer organization delete --name org33[0;36;49m [command #60][0m
[0;34;49mCould not delete the organization:
404 Not Found[0m
[0;31;49m[FAIL] [0mreturns ok
user
hammer user delete --login some_user33[0;36;49m [command #61][0m
[0;32;49m[ OK ] [0mreturns ok
architecture
hammer architecture delete --name arch33[0;36;49m [command #62][0m
[0;32;49m[ OK ] [0mreturns ok
operating system
hammer os delete --id 2[0;36;49m [command #63][0m
[0;32;49m[ OK ] [0mreturns ok
partition table
hammer partition-table delete --name ptable33[0;36;49m [command #64][0m
[0;32;49m[ OK ] [0mreturns ok
hardware model
hammer model delete --name model33[0;36;49m [command #65][0m
[0;32;49m[ OK ] [0mreturns ok
template
hammer template delete --name tpl33[0;36;49m [command #66][0m
[0;32;49m[ OK ] [0mreturns ok
simple listing
hammer architecture list[0;36;49m [command #67][0m
[0;32;49m[ OK ] [0mreturns ok
hammer compute-resource list[0;36;49m [command #68][0m
[0;32;49m[ OK ] [0mreturns ok
hammer domain list[0;36;49m [command #69][0m
[0;32;49m[ OK ] [0mreturns ok
hammer environment list[0;36;49m [command #70][0m
[0;32;49m[ OK ] [0mreturns ok
hammer fact list[0;36;49m [command #71][0m
[0;32;49m[ OK ] [0mreturns ok
hammer global-parameter list[0;36;49m [command #72][0m
[0;32;49m[ OK ] [0mreturns ok
hammer host list[0;36;49m [command #73][0m
[0;32;49m[ OK ] [0mreturns ok
hammer hostgroup list[0;36;49m [command #74][0m
[0;32;49m[ OK ] [0mreturns ok
hammer location list[0;36;49m [command #75][0m
[0;34;49mThe server does not support such operation.[0m
[0;31;49m[FAIL] [0mreturns ok
hammer medium list[0;36;49m [command #76][0m
[0;32;49m[ OK ] [0mreturns ok
hammer model list[0;36;49m [command #77][0m
[0;32;49m[ OK ] [0mreturns ok
hammer organization list[0;36;49m [command #78][0m
[0;34;49mThe server does not support such operation.[0m
[0;31;49m[FAIL] [0mreturns ok
hammer os list[0;36;49m [command #79][0m
[0;32;49m[ OK ] [0mreturns ok
hammer partition-table list[0;36;49m [command #80][0m
[0;32;49m[ OK ] [0mreturns ok
hammer proxy list[0;36;49m [command #81][0m
[0;32;49m[ OK ] [0mreturns ok
hammer puppet-class list[0;36;49m [command #82][0m
[0;32;49m[ OK ] [0mreturns ok
hammer report list[0;36;49m [command #83][0m
[0;32;49m[ OK ] [0mreturns ok
hammer sc-param list[0;36;49m [command #84][0m
[0;32;49m[ OK ] [0mreturns ok
hammer subnet list[0;36;49m [command #85][0m
[0;32;49m[ OK ] [0mreturns ok
hammer template list[0;36;49m [command #86][0m
[0;32;49m[ OK ] [0mreturns ok
hammer user list[0;36;49m [command #87][0m
[0;32;49m[ OK ] [0mreturns ok
--------------------------------------------------------------------------------
[0;31;49mCommands: 10 out of 87 failed[0m
[0;31;49mTests: 10 out of 162 failed[0m