{"id":153,"date":"2017-09-21T10:07:01","date_gmt":"2017-09-21T10:07:01","guid":{"rendered":"https:\/\/arxcruz.net\/?p=153"},"modified":"2017-09-21T10:11:07","modified_gmt":"2017-09-21T10:11:07","slug":"debugging-tempest","status":"publish","type":"post","link":"https:\/\/arxcruz.net\/index.php\/2017\/09\/21\/debugging-tempest\/","title":{"rendered":"Debugging tempest"},"content":{"rendered":"

Introduction<\/h1>\n

One of the common tasks when you’re working on TripleO is to debug tempest when you found some failures. This is very important because once you found an error, you need to provide all the information needed when you’re gonna fill a bug, or you can even actually, find the root cause of the problem, and fix it yourself!
\nIt’s also important to know when a failure in tempest is actually a bug, when is a miss configuration in some .conf file, timeouts, etc.
\nIn this post I’ll try to dig into all these aspects when you’re debugging tempest. But before, we need to make some steps, like setup our OpenStack environment, download tempest, setup the environment and then hands on in some cases.<\/p>\n

Installing TripleO<\/h1>\n

I’ll not cover here how to install TripleO, you can follow the official documentation here<\/a>.<\/p>\n

Preparing the environment<\/h2>\n

Once you have TripleO installed, we need to access the undercloud, and from there, we need to setup tempest environment. You can use both tempest upstream, or the tempest RDO package. We will use the last one.<\/p>\n

Usually, TripleO install everything with the user stack, and in this directory we can find all the scripts and images used to install our environment.<\/p>\n

\r\nssh stack@undercloud\r\n<\/pre>\n
Now, we need to source the credentials that TripleO provide to us:<\/p>\n
\r\nsource \/home\/stack\/overcloudrc.v3\r\n<\/pre>\n
There’s an overcloudrc for v2 api, however, it’s going to be deprecated soon.<\/p>\n
You also need to have a public network available. If you don’t have it, the follow script can do that for you:<\/p>\n
\r\n#!\/bin\/bash\r\n\r\nfor i in $(neutron floatingip-list -c id -f value)\r\ndo\r\nneutron floatingip-disassociate $i\r\nneutron floatingip-delete $i\r\ndone\r\nfor i in $(neutron router-list -c id -f value); do neutron router-gateway-clear $i; done\r\nfor r in $(neutron router-list -c id -f value); do\r\nfor p in $(neutron router-port-list $r -c id -f value); do\r\nneutron router-interface-delete $r port=$p || true\r\ndone\r\ndone\r\nfor i in $(neutron router-list -c id -f value); do neutron router-delete $i; done\r\nfor i in $(neutron port-list -c id -f value); do neutron port-delete $i; done\r\nfor i in $(neutron net-list -c id -f value); do neutron net-delete $i; done\r\n\r\nneutron net-create public --router:external=True \\\r\n--provider:network_type flat \\\r\n--provider:physical_network datacentre\r\n\r\npublic_net_id=$(neutron net-show public -f value -c id)\r\n\r\nneutron subnet-create --name ext-subnet \\\r\n--allocation-pool \\\r\nstart=192.168.24.100,end=192.168.24.120 \\\r\n--disable-dhcp \\\r\n--gateway 192.168.24.1 \\\r\npublic 192.168.24.0\/24\r\n<\/pre>\n
Set the start and end ip for your external subnet according your network environment.<\/p>\n
Install the openstack-tempest, python-junitxml and python-tempestconf:<\/p>\n
\r\nsudo yum -y install openstack-tempest python-junitxml python-tempestconf\r\n<\/pre>\n
Initialize tempest in a new directory (you don’t need to mkdir this, tempest will do that for you)<\/p>\n
\r\ntempest init \/home\/stack\/tempest\r\ncd \/home\/stack\/tempest\r\n<\/pre>\n
The output should be something very straightforward:<\/p>\n
\r\n2017-09-21 08:01:07.215 1049 INFO tempest [-] Using tempest config file \/etc\/tempest\/tempest.conf\r\n<\/pre>\n
Tempestconf<\/h2>\n
Tempestconf is a python script created to make it easy to create a tempest.conf file. If you ever worked with tempest, you know it’s pretty hard to have everything setup properly manually, and tempestconf tries to detect as much as it can from which services are running in your environment, what you have enabled, and what you don’t, etc.
\nAssuming you have sourced the overcloudrc.v3 and you have executed the script to create your public network, you can run the following command:<\/p>\n
\r\ndiscover-tempest-config --out etc\/tempest.conf \\\r\n--network-id $public_net_id \\\r\n--deployer-input ~\/tempest-deployer-input.conf \\\r\n--debug --create \\\r\nidentity.uri $OS_AUTH_URL \\\r\nauth.admin_password $OS_PASSWORD \\\r\nauth.admin_username $OS_USERNAME \\\r\nauth.use_dynamic_credentials true \\\r\ncompute-feature-enabled.attach_encrypted_volume False \\\r\nnetwork.tenant_network_cidr 192.168.0.0\/24 \\\r\ncompute.build_timeout 500 \\\r\nvolume-feature-enabled.api_v1 False \\\r\nvalidation.image_ssh_user cirros \\\r\nvalidation.ssh_user cirros \\\r\nnetwork.build_timeout 500 \\\r\nvolume.build_timeout 500 \\\r\norchestration.stack_owner_role heat_stack_owner\r\n<\/pre>\n
As you can see, tempest can get information from other files, for example the tempest-deployer-input.conf that is created by TripleO with some information like the region, volume backend being used, etc.
\nYou can also manually set some configs, following the pattern ‘section.config_name value’. For example, in the section validation we are setting the ssh_user option to cirros. In the tempest.conf it will show something like this:<\/p>\n
\r\n[validation]\r\nssh_user = cirros\r\n\r\nOnce the script finishes you will have a tempest.conf in your \/home\/stack\/tempest\/etc\/ directory that with something like this:\r\n\r\n[DEFAULT]\r\ndebug = true\r\nuse_stderr = false\r\nlog_file = tempest.log\r\n\r\n[auth]\r\ntempest_roles = _member_\r\nadmin_username = admin\r\nadmin_project_name = admin\r\nadmin_domain_name = Default\r\nuse_dynamic_credentials = true\r\nadmin_password = Tudv4RGAhwnJq8UPvgbamUAkY\r\n\r\n[compute]\r\nimage_ssh_user = cirros\r\nregion = regionOne\r\nbuild_timeout = 500\r\nflavor_ref = d8668b47-5d68-4729-b3f6-5eed67eb50fe\r\nflavor_ref_alt = 313dfd3d-059a-4dc2-a6da-6ebc32cfe546\r\nimage_ref = 31edf0e4-eed4-4b09-96cd-1f8c9c78887d\r\nimage_ref_alt = 7943a836-218c-4902-bf4a-1a36ca52010c\r\n\r\n[identity]\r\nusername = demo\r\ntenant_name = demo\r\npassword = secrete\r\nalt_username = alt_demo\r\nalt_tenant_name = alt_demo\r\nalt_password = secrete\r\nadmin_username = admin\r\nadmin_tenant_name = admin\r\nadmin_domain_name = Default\r\ndisable_ssl_certificate_validation = true\r\nregion = regionOne\r\nadmin_password = Tudv4RGAhwnJq8UPvgbamUAkY\r\nuri = http:\/\/192.168.24.7:5000\/v2.0\r\nauth_version = v3\r\nuri_v3 = http:\/\/192.168.24.7:5000\/v3\r\nadmin_tenant_id = a1e4e1ee88804a8cacc0796757a5a0c1\r\n\r\n[object-storage]\r\noperator_role = SwiftOperator\r\nregion = regionOne\r\n\r\n[data-processing]\r\n\r\n[orchestration]\r\nstack_owner_role = heat_stack_owner\r\nregion = regionOne\r\n\r\n[scenario]\r\nimg_dir = etc\r\n\r\n[oslo_concurrency]\r\nlock_path = \/tmp\r\n\r\n[volume-feature-enabled]\r\nbootable = true\r\nvolume_services = true\r\napi_v1 = False\r\napi_v2 = True\r\napi_v3 = True\r\napi_extensions =\r\n\r\n[compute-feature-enabled]\r\nlive_migration = false\r\nlive_migrate_paused_instances = true\r\npreserve_ports = true\r\nconsole_output = false\r\nattach_encrypted_volume = False\r\napi_extensions = NMN,OS-DCF,OS-EXT-AZ,OS-EXT-IMG-SIZE,OS-EXT-IPS,OS-EXT-IPS-MAC,OS-EXT-SRV-ATTR,OS-EXT-STS,OS-FLV-DISABLED,OS-FLV-EXT-DATA,OS-SCH-HNT,OS-SRV-USG,os-access-ips,os-admin-actions,os-admin-password,os-agents,os-aggregates,os-assisted-volume-snapshots,os-attach-interfaces,os-availability-zone,os-baremetal-ext-status,os-baremetal-nodes,os-block-device-mapping,os-block-device-mapping-v2-boot,os-cell-capacities,os-cells,os-certificates,os-cloudpipe,os-cloudpipe-update,os-config-drive,os-console-auth-tokens,os-console-output,os-consoles,os-create-backup,os-create-server-ext,os-deferred-delete,os-evacuate,os-extended-evacuate-find-host,os-extended-floating-ips,os-extended-hypervisors,os-extended-networks,os-extended-quotas,os-extended-rescue-with-image,os-extended-services,os-extended-services-delete,os-extended-status,os-extended-volumes,os-fixed-ips,os-flavor-access,os-flavor-extra-specs,os-flavor-manage,os-flavor-rxtx,os-flavor-swap,os-floating-ip-dns,os-floating-ip-pools,os-floating-ips,os-floating-ips-bulk,os-fping,os-hide-server-addresses,os-hosts,os-hypervisor-status,os-hypervisors,os-instance-actions,os-instance_usage_audit_log,os-keypairs,os-lock-server,os-migrate-server,os-migrations,os-multiple-create,os-networks,os-networks-associate,os-pause-server,os-personality,os-preserve-ephemeral-rebuild,os-quota-class-sets,os-quota-sets,os-rescue,os-security-group-default-rules,os-security-groups,os-server-diagnostics,os-server-external-events,os-server-group-quotas,os-server-groups,os-server-list-multi-status,os-server-password,os-server-sort-keys,os-server-start-stop,os-services,os-shelve,os-simple-tenant-usage,os-suspend-server,os-tenant-networks,os-used-limits,os-used-limits-for-admin,os-user-data,os-user-quotas,os-virtual-interfaces,os-volume-attachment-update,os-volumes\r\n\r\n[network-feature-enabled]\r\nipv6_subnet_attributes = true\r\napi_extensions = default-subnetpools,qos,network-ip-availability,network_availability_zone,auto-allocated-topology,ext-gw-mode,binding,agent,subnet_allocation,l3_agent_scheduler,tag,external-net,standard-attr-tag,flavors,net-mtu,availability_zone,qos-default,quotas,revision-if-match,l3-ha,provider,multi-provider,quota_details,address-scope,trunk,extraroute,net-mtu-writable,subnet-service-types,standard-attr-timestamp,service-type,qos-rule-type-details,l3-flavors,port-security,extra_dhcp_opt,standard-attr-revisions,pagination,sorting,security-group,dhcp_agent_scheduler,router_availability_zone,rbac-policies,qos-bw-limit-direction,tag-ext,standard-attr-description,router,allowed-address-pairs,project-id,trunk-details\r\n\r\n[validation]\r\nimage_ssh_user = cirros\r\nssh_user = cirros\r\n\r\n[image]\r\nregion = regionOne\r\n\r\n[network]\r\nregion = regionOne\r\ntenant_network_cidr = 192.168.0.0\/24\r\nbuild_timeout = 500\r\npublic_network_id = 9c5f91dd-118e-4637-a3cf-809f672a3c38\r\nfloating_network_name = public\r\n\r\n[volume]\r\nbackend1_name = tripleo_iscsi\r\nregion = regionOne\r\nbuild_timeout = 500\r\n\r\n[service_available]\r\nglance = True\r\nmanila = False\r\ncinder = True\r\nswift = True\r\nsahara = False\r\npanko = True\r\nnova = True\r\nneutron = True\r\ntrove = False\r\nceilometer = True\r\nironic = False\r\nheat = True\r\nzaqar = True\r\ngnocchi = True\r\naodh = True\r\naodh_plugin = True\r\nhorizon = False\r\n\r\n[image-feature-enabled]\r\napi_v1 = False\r\napi_v2 = True\r\n\r\n[identity-feature-enabled]\r\napi_v2 = True\r\napi_v3 = True\r\napi_extensions = https:,OS-REVOKE,OS-FEDERATION,OS-KSCRUD,OS-SIMPLE-CERT,OS-OAUTH1,OS-EC2\r\nforbid_global_implied_dsr = True\r\n\r\n[object-storage-feature-enabled]\r\ndiscoverable_apis = formpost,bulk_delete,versioned_writes,container_quotas,tempurl,bulk_upload,ratelimit,slo,account_quotas,staticweb\r\n\r\n[dashboard]\r\ndashboard_url = http:\/\/192.168.24.7\/dashboard\/\r\nlogin_url = http:\/\/192.168.24.7\/dashboard\/auth\/login\/\r\n<\/pre>\n
Testing if everything is working properly<\/p>\n
Once we have finished setup tempest, let’s see if everything works properly, you can check if we have some plugins for tempest installed:<\/p>\n
\r\ntempest list-plugins\r\n\r\n+------------------+---------------------------------------------------------+\r\n| Name | EntryPoint |\r\n+------------------+---------------------------------------------------------+\r\n| barbican_tests | barbican_tempest_plugin.plugin:BarbicanTempestPlugin |\r\n| zaqar_tests | zaqar.tests.tempest_plugin.plugin:ZaqarTempestPlugin |\r\n| ceilometer_tests | ceilometer.tests.tempest.plugin:CeilometerTempestPlugin |\r\n| gnocchi_tests | gnocchi.tempest.plugin:GnocchiTempestPlugin |\r\n| aodh_tests | aodh.tests.tempest.plugin:AodhTempestPlugin |\r\n| aws_tests | ec2api.tests.functional.plugin:AWSTempestPlugin |\r\n+------------------+---------------------------------------------------------+\r\n<\/pre>\n
And also list the tests:<\/p>\n
\r\nostestr -l\r\n<\/pre>\n
\r\nrunning=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \\\r\nOS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \\\r\nOS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \\\r\n${PYTHON:-python} -m subunit.run discover -t \/usr\/lib\/python2.7\/site-packages\/tempest \/usr\/lib\/python2.7\/site-packages\/tempest\/test_discover --list\r\ntempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_create_agent[id-1fc6bdc8-0b6d-4cc7-9f30-9b04fabe5b90]\r\ntempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_delete_agent[id-470e0b89-386f-407b-91fd-819737d0b335]\r\ntempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents[id-6a326c69-654b-438a-80a3-34bcc454e138]\r\ntempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents_with_filter[id-eabadde4-3cd7-4ec4-a4b5-5a936d2d4408]\r\ntempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_update_agent[id-dc9ffd51-1c50-4f0e-a820-ae6d2a568a9e]\r\ntempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_create_server_with_az[id-96be03c7-570d-409c-90f8-e4db3c646996]\r\ntempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_get_details[id-eeef473c-7c52-494d-9f09-2ed7fc8fc036]\r\ntempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_list[id-7f6a1cc5-2446-4cdb-9baa-b6ae0a919b72]\r\n...\r\n<\/pre>\n
It’s a huge list!<\/p>\n
Running tempest<\/p>\n
Well, in order to have something to debug, we must run some tests, and here we can first save the actual state of the cloud:<\/p>\n
\r\ntempest cleanup --init-saved-state\r\n<\/pre>\n
This will create a saved_state.json file with the information about how our environment was before we ran tempest. This is good because if some test fails in the tearDownClass step, it might leave some dirty behind, and we can cleanup later.<\/p>\n
Smoke<\/p>\n
Tempest has a set of tests called smoke, these tests are supposed to be fast and can verify the sanity of your system, and that’s what we going to run here:<\/p>\n
\r\nostestr smoke\r\n<\/pre>\n
Debugging tempest<\/p>\n
And now what everybody was expecting!
\nNow that we ran tempest and we have our results, we can start to debug these errors.<\/p>\n
Type of errors<\/p>\n
In my opinion, tempest have several types of errors, there are some errors that it’s actually a bug, and you need to report, others are errors due miss-configuration files, others because some timeout we will try to cover mostly of the types here<\/p>\n
Getting the list of errors<\/p>\n
When tempest finishes to run, it shows your a report with the number of tests executed, the success and failures, like this one:<\/p>\n
\r\n======\r\nTotals\r\n======\r\nRan: 1522 tests in 3417.0000 sec.\r\n- Passed: 1408\r\n- Skipped: 87\r\n- Expected Fail: 0\r\n- Unexpected Success: 0\r\n- Failed: 27\r\nSum of execute time for each test: 7235.4187 sec.\r\n\r\n==============\r\nWorker Balance\r\n==============\r\n- Worker 0 (367 tests) =&amp;gt; 0:45:38.887316\r\n- Worker 1 (384 tests) =&amp;gt; 0:38:19.733741\r\n- Worker 2 (325 tests) =&amp;gt; 0:56:47.206368\r\n- Worker 3 (446 tests) =&amp;gt; 0:46:33.747285\r\n\r\n<\/pre>\n
But that’s not enough to us, we need to get the list of errors, for that, we can use the testr tool to get the failing errors, for example, when you execute the following command:<\/p>\n
\r\ntestr failing |grep FAIL | sed -e 's\/\\[.*\\].*\/\/g' -e 's\/FAIL\\: \/\/'\r\n<\/pre>\n
We will see something like this:<\/p>\n
\r\ntempest.api.compute.certificates.test_certificates.CertificatesV2TestJSON.test_create_root_certificate\r\ntempest.api.compute.certificates.test_certificates.CertificatesV2TestJSON.test_get_root_certificate\r\nsetUpClass (tempest.api.orchestration.stacks.test_nova_keypair_resources.NovaKeyPairResourcesYAMLTest)\r\nsetUpClass (tempest.api.orchestration.stacks.test_swift_resources.SwiftResourcesTestJSON)\r\ntempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container\r\nsetUpClass (tempest.api.orchestration.stacks.test_templates.TemplateYAMLTestJSON)\r\nsetUpClass (tempest.api.orchestration.stacks.test_templates_negative.TemplateAWSNegativeTestJSON)\r\nsetUpClass (tempest.api.orchestration.stacks.test_volumes.CinderResourcesTest)\r\nsetUpClass (tempest.api.orchestration.stacks.test_limits.TestServerStackLimits)\r\nsetUpClass (tempest.api.orchestration.stacks.test_non_empty_stack.StacksTestJSON)\r\nsetUpClass (tempest.api.orchestration.stacks.test_resource_types.ResourceTypesTest)\r\nsetUpClass (tempest.api.orchestration.stacks.test_soft_conf.TestSoftwareConfig)\r\nsetUpClass (tempest.api.orchestration.stacks.test_templates_negative.TemplateYAMLNegativeTestJSON)\r\ntempest.api.volume.admin.test_volumes_backup.VolumesBackupsV1Test.test_volume_backup_create_get_detailed_list_restore_delete\r\ntempest.api.volume.admin.test_volumes_backup.VolumesBackupsV1Test.test_volume_backup_export_import\r\ntempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_cryptsetup\r\ntempest.api.volume.admin.test_volumes_backup.VolumesBackupsV2Test.test_volume_backup_create_get_detailed_list_restore_delete\r\ntempest.api.volume.admin.test_volumes_backup.VolumesBackupsV2Test.test_volume_backup_export_import\r\nsetUpClass (tempest.api.orchestration.stacks.test_environment.StackEnvironmentTest)\r\nsetUpClass (tempest.api.orchestration.stacks.test_nova_keypair_resources.NovaKeyPairResourcesAWSTest)\r\nsetUpClass (tempest.api.orchestration.stacks.test_stacks.StacksTestJSON)\r\nsetUpClass (tempest.api.orchestration.stacks.test_templates.TemplateAWSTestJSON)\r\ntempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_luks\r\ntempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_reboot\r\ntempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario\r\ntempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic\r\ntempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_multiple_security_groups\r\n<\/pre>\n
setUpClass<\/p>\n
You have noticed that some failures start with this setUpClass, this means, that the test haven’t even started because some error were found when the test was setting whatever the test do (create VM’s, networks, etc).<\/p>\n
tempest.something.something_else.MyClassV2.test_that_fails<\/p>\n
When you see some failure like that, it means that the test pass from the setUpClass phase, and actually tried to run, and it fails for some reason, don’t worry, we will debug it.<\/p>\n
tearDownClass<\/p>\n
Some tests fails when it tries to clean the test, for example, in the setUpClass phase, it creates a VM, but in the tearDownClass the VM wasn’t able to be deleted, it’s rare, but it can happen.<\/p>\n
The debug!<\/p>\n
Let’s try to debug something.<\/p>\n
Re-running one particular test<\/p>\n
The easiest way to do that is using python testtools with the following command:<\/p>\n
\r\npython -m testtools.run tempest.test.that.i.want.to.run\r\n<\/pre>\n
Why not use ostestr to run that particular test? Because, we are going to use a tool called pudb, and you can’t use it with ostestr.<\/p>\n
Was it a timeout?<\/p>\n
The first thing we do when we are debugging, is to re-run the test again to be sure that wasn’t a failure based on some timeout, sometimes the machine or network are struggling and we have a timeout. Tempest have a timeout that is configured in etc\/tempest.conf that can, for example, tell that if a VM doesn’t reach the status ACTIVE in 60 seconds, mark the test as a failure, but if you’re running tests in parallel, the machine might take more time to reach ACTIVE because all resources are full for example. So, it’s always a good idea re-run the test, and also, rerunning the test will show the output of the failure, which is a good source of information.<\/p>\n
So, let’s see a few examples of failures and how we debug it:<\/p>\n
Let’s get the first failure test tempest.api.compute.certificates.test_certificates.CertificatesV2TestJSON.test_create_root_certificate<\/p>\n
\r\npython -m testtools.run tempest.api.compute.certificates.test_certificates.CertificatesV2TestJSON.test_create_root_certificate\r\n<\/pre>\n
And after a while, here’s the output:<\/p>\n
\r\nTests running...\r\n======================================================================\r\nFAIL: tempest.api.compute.certificates.test_certificates.CertificatesV2TestJSON.test_create_root_certificate[id-c070a441-b08e-447e-a733-905909535b1b]\r\n----------------------------------------------------------------------\r\npythonlogging:'': {{{\r\n2016-06-15 10:18:15,733 28198 INFO [tempest.lib.common.rest_client] Request (CertificatesV2TestJSON:test_create_root_certificate): 504 POST http:\/\/10.0.0.101:8774\/v2.1\/832848e6175148aa874dbb2a891cd02a\/os-certificates 60.004s\r\n2016-06-15 10:18:15,734 28198 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application\/json', 'Accept': 'application\/json', 'X-Auth-Token': ''}\r\nBody: None\r\nResponse - Headers: {'status': '504', 'connection': 'close', 'content-type': 'text\/html', 'content-location': 'http:\/\/10.0.0.101:8774\/v2.1\/832848e6175148aa874dbb2a891cd02a\/os-certificates', 'cache-control': 'no-cache'}\r\nBody:\r\n\r\n&lt;h1&gt;504 Gateway Time-out&lt;\/h1&gt;\r\n\r\nThe server didn't respond in time.\r\n\r\n}}}\r\n\r\nTraceback (most recent call last):\r\nFile "tempest\/api\/compute\/certificates\/test_certificates.py", line 34, in test_create_root_certificate\r\nbody = self.certificates_client.create_certificate()['certificate']\r\nFile "tempest\/lib\/services\/compute\/certificates_client.py", line 35, in create_certificate\r\nresp, body = self.post(url, None)\r\nFile "tempest\/lib\/common\/rest_client.py", line 270, in post\r\nreturn self.request('POST', url, extra_headers, headers, body, chunked)\r\nFile "tempest\/lib\/services\/compute\/base_compute_client.py", line 53, in request\r\nmethod, url, extra_headers, headers, body, chunked)\r\nFile "tempest\/lib\/common\/rest_client.py", line 664, in request\r\nresp, resp_body)\r\nFile "tempest\/lib\/common\/rest_client.py", line 832, in _error_checker\r\nresp=resp)\r\ntempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received\r\nDetails: 504\r\n\r\nRan 1 test in 61.744s\r\nFAILED (failures=1)\r\n<\/pre>\n
Notice the lines 9-16. The first thing that we can notice is that we got a Gateway Time-out.
\nChecking the name of the test, it’s trying to create a root certificate, and check the content-location: http:\/\/10.0.0.101:8774\/v2.1\/832848e6175148aa874dbb2a891cd02a\/os-certificates
\nIt’s hard to figure out, but in this case, os-certificates it’s from a service, openstack-nova-cert, so, the first thing that we need to check is if this service is up and running in our controller, so, we need to ssh to the controller machine and check it.
\nYou already know the IP address that is 10.0.0.101, and the user is heat-admin<\/p>\n
\r\nssh heat-admin@10.0.0.101\r\n<\/pre>\n
In order to ssh works, you need to be logged as the ‘stack’ user<\/p>\n
Now in the controller machine, let’s see the status of the services:<\/p>\n
\r\nsystemctl status openstack-*\r\n<\/pre>\n
This will show a list of all services, and you can notice this:<\/p>\n
\r\n\u25cf openstack-nova-cert.service - OpenStack Nova Cert Server\r\nLoaded: loaded (\/usr\/lib\/systemd\/system\/openstack-nova-cert.service; enabled; vendor preset: disabled)\r\n<\/pre>\n
As you can see, the openstack-nova-cert service is INACTIVE!!!
\nLet’s start it<\/p>\n
\r\nsudo systemctl start openstack-nova-cert\r\n<\/pre>\n
And check if the service was successfuly sarted:<\/p>\n
\r\nsystemctl status openstack-nova-cert\r\nsystemctl status openstack-nova-cert\r\n<\/pre>\n
And here’s the output:<\/p>\n
\r\n\u25cf openstack-nova-cert.service - OpenStack Nova Cert Server\r\nLoaded: loaded (\/usr\/lib\/systemd\/system\/openstack-nova-cert.service; disabled; vendor preset: disabled)\r\nActive: active (running) since Qua 2017-09-21 14:46:43 UTC; 51s ago\r\nMain PID: 6985 (nova-cert)\r\nCGroup: \/system.slice\/openstack-nova-cert.service\r\n\u2514\u25006985 \/usr\/bin\/python2 \/usr\/bin\/nova-cert\r\n<\/pre>\n
Now, let’s go back to our undercloud, and try again run the test. Please remember to activate the virtualenv again.<\/p>\n
\r\npython -m testtools.run tempest.api.compute.certificates.test_certificates.CertificatesV2TestJSON.test_create_root_certificate\r\n<\/pre>\n
And now we have a different output:<\/p>\n
\r\nTests running...\r\n\r\nRan 1 test in 1.153s\r\nOK\r\n<\/pre>\n
And congratulations! You have debugged your first test in tempest!
\nBut this was a easy one, let’s go for something more challenging<\/p>\n
More challenge stuff<\/p>\n
Let’s get the next failure tempest.api.orchestration.stacks.test_nova_keypair_resources.NovaKeyPairResourcesYAMLTest<\/p>\n
\r\npython -m testtools.run tempest.api.orchestration.stacks.test_nova_keypair_resources.NovaKeyPairResourcesYAMLTest\r\n<\/pre>\n
Output:<\/p>\n
\r\nTests running...\r\n======================================================================\r\nERROR: setUpClass (tempest.api.orchestration.stacks.test_nova_keypair_resources.NovaKeyPairResourcesYAMLTest)\r\n----------------------------------------------------------------------\r\nTraceback (most recent call last):\r\nFile "tempest\/test.py", line 278, in setUpClass\r\nsix.reraise(etype, value, trace)\r\nFile "tempest\/test.py", line 266, in setUpClass\r\ncls.setup_credentials()\r\nFile "tempest\/api\/orchestration\/base.py", line 40, in setup_credentials\r\ncls.os = cls.get_client_manager(roles=[stack_owner_role])\r\nFile "tempest\/test.py", line 536, in get_client_manager\r\ncreds = cred_provider.get_creds_by_roles(**params)\r\nFile "tempest\/common\/dynamic_creds.py", line 323, in get_creds_by_roles\r\nreturn self.get_credentials(roles)\r\nFile "tempest\/common\/dynamic_creds.py", line 286, in get_credentials\r\ncredentials = self._create_creds(roles=credential_type)\r\nFile "tempest\/common\/dynamic_creds.py", line 157, in _create_creds\r\nself.creds_client.assign_user_role(user, project, role)\r\nFile "tempest\/common\/cred_client.py", line 72, in assign_user_role\r\nraise lib_exc.NotFound(msg)\r\ntempest.lib.exceptions.NotFound: Object not found\r\nDetails: No "heat_stack_owner" role found\r\n\r\nRan 0 tests in 4.887s\r\nFAILED (failures=1)\r\n<\/pre>\n
Now, things are getting interested here, it wasn’t a timeout, it wasn’t some service that weren’t enabled, but you can see in the details that was a role that wasn’t founded.
\nso, let’s verify first which roles we have in keystone:<\/p>\n
\r\nsource \/home\/stack\/overcloudrc.v3\r\nopenstack role list\r\n<\/pre>\n
\r\n+----------------------------------+------------------+\r\n| ID | Name |\r\n+----------------------------------+------------------+\r\n| 3d7ddc928c3044988b204730a3087d0d | swiftoperator |\r\n| 7608609651af49369aa68510715a3507 | admin |\r\n| 9fe2ff9ee4384b1894a90878d3e92bab | _member_ |\r\n| cc2203d86a2a496c9b62876bbe8ffaa0 | heat_stack_user |\r\n| cfd9df98de374b678f756d1ed8dd3662 | ResellerAdmin |\r\n+----------------------------------+------------------+\r\n<\/pre>\n
So, as you can see, the role is heat_stack_user, instead of heat_stack_owner, but why tempest is trying to use heat_stack_owner?
\nLet’s see:<\/p>\n
\r\ncd \/usr\/lib\/python2.7\/site-packages\/tempest\r\ngrep heat_stack_owner * -R\r\n<\/pre>\n
\r\nconfig.py: cfg.StrOpt('stack_owner_role', default='heat_stack_owner',\r\n<\/pre>\n
Now, that’s interesting, if you notice tempest\/config.py, the default role is heat_stack_owner.
\nTempest have default values for everything that you can set in etc\/tempest.conf. If there’s nothing set in tempest.conf file, tempest will use the default one, so we have here two options: creat
\ne a heat_stack_owner role, or change\/add the stack_owner_role option in our etc\/tempest.conf. I believe the second option is better so let’s do it.
\nHow do we know which section we need to add it?
\nIf you open the tempest\/config.py and look for the heat_stack_owner, you will see that it’s under OrchestrationGroup:<\/p>\n
\r\norchestration_group = cfg.OptGroup(name='orchestration',\r\ntitle='Orchestration Service Options')\r\n\r\nOrchestrationGroup = [\r\ncfg.StrOpt('catalog_type',\r\ndefault='orchestration',\r\nhelp="Catalog type of the Orchestration service."),\r\ncfg.StrOpt('region',\r\ndefault='',\r\nhelp="The orchestration region name to use. If empty, the "\r\n"value of identity.region is used instead. If no such "\r\n"region is found in the service catalog, the first found "\r\n"one is used."),\r\ncfg.StrOpt('endpoint_type',\r\ndefault='publicURL',\r\nchoices=['public', 'admin', 'internal',\r\n'publicURL', 'adminURL', 'internalURL'],\r\nhelp="The endpoint type to use for the orchestration service."),\r\ncfg.StrOpt('stack_owner_role', default='heat_stack_owner',\r\n...\r\n(orchestration_group, OrchestrationGroup),\r\n<\/pre>\n
OrchestrationGroup became orechestration in etc\/tempest.conf. So, let’s go there and add the following line under [orchestration] in etc\/tempest.conf<\/p>\n
\r\nstack_owner_role = heat_stack_user\r\n<\/pre>\n
Let’s try again?<\/p>\n
\r\npython -m testtools.run tempest.api.orchestration.stacks.test_nova_keypair_resources.NovaKeyPairResourcesYAMLTest\r\n<\/pre>\n
And the output:<\/p>\n
\r\nTests running...\r\n======================================================================\r\nERROR: setUpClass (tempest.api.orchestration.stacks.test_nova_keypair_resources.NovaKeyPairResourcesYAMLTest)\r\n----------------------------------------------------------------------\r\nTraceback (most recent call last):\r\nFile "tempest\/test.py", line 278, in setUpClass\r\nsix.reraise(etype, value, trace)\r\nFile "tempest\/test.py", line 271, in setUpClass\r\ncls.resource_setup()\r\nFile "tempest\/api\/orchestration\/stacks\/test_nova_keypair_resources.py", line 35, in resource_setup\r\n'KeyPairName2': cls.stack_name + '_2'\r\nFile "tempest\/api\/orchestration\/base.py", line 77, in create_stack\r\nfiles=files)\r\nFile "tempest\/services\/orchestration\/json\/orchestration_client.py", line 56, in create_stack\r\nresp, body = self.post(uri, headers=headers, body=body)\r\nFile "tempest\/lib\/common\/rest_client.py", line 270, in post\r\nreturn self.request('POST', url, extra_headers, headers, body, chunked)\r\nFile "tempest\/lib\/common\/rest_client.py", line 664, in request\r\nresp, resp_body)\r\nFile "tempest\/lib\/common\/rest_client.py", line 757, in _error_checker\r\nraise exceptions.Forbidden(resp_body, resp=resp)\r\ntempest.lib.exceptions.Forbidden: Forbidden\r\nDetails: {u'title': u'Forbidden', u'explanation': u'Access was denied to this resource.', u'code': 403, u'error': {u'message': u'You are not authorized to use create.', u'traceback': None, u'type': u'Forbidden'}}\r\n\r\nRan 0 tests in 9.718s\r\nFAILED (failures=1)\r\n<\/pre>\n
And now we have a different failure, in this case, the role doesn’t have the rights to add these resources.
\nWell, we can then use _member_ role, since this was the default role for everything in tempest before they start to create roles separately. Let’s change again our etc\/tempest.conf in [orchestration] section:<\/p>\n
\r\nstack_owner_role = _member_\r\n<\/pre>\n
Running again…<\/p>\n
\r\npython -m testtools.run tempest.api.orchestration.stacks.test_nova_keypair_resources.NovaKeyPairResourcesYAMLTest\r\n<\/pre>\n
And the output:<\/p>\n
\r\nTests running...\r\n\r\nRan 2 tests in 16.784s\r\nOK\r\n<\/pre>\n
The pudb in action<\/h2>\n
Here is where things get interesting, let’s check the next failure in our list:<\/p>\n
\r\npython -m testtools.run tempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container\r\n<\/pre>\n
\r\nTests running...\r\n\r\n======================================================================\r\nFAIL: tempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container[id-a151e158-dcbf-4a1f-a1e7-46cd65895a6f]\r\n----------------------------------------------------------------------\r\npythonlogging:'': {{{\r\n2016-06-16 07:15:36,531 15907 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_versioned_container): 200 POST http:\/\/10.0.0.101:5000\/v2.0\/tokens\r\n2016-06-16 07:15:36,531 15907 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application\/json', 'Accept': 'application\/json'}\r\nBody:\r\nResponse - Headers: {'status': '200', 'content-length': '4540', 'content-location': 'http:\/\/10.0.0.101:5000\/v2.0\/tokens', 'vary': 'X-Auth-Token,Accept-Encoding', 'server': 'Apache\/2.4.6 (Red Hat Enterprise Linux)', 'connection': 'close', 'date': 'Thu, 16 Jun 2016 11:15:36 GMT', 'content-type': 'application\/json', 'x-openstack-request-id': 'req-f79820a7-d01a-4ac2-a6ea-ac76f63f6070'}\r\nBody: {"access": {"token": {"issued_at": "2016-06-16T11:15:36.517581Z", "expires": "2016-06-16T12:15:36Z", "id": "28fba048ed524fdcb5309a6cd2e6975c", "tenant": {"description": "tempest-ContainerTest-272815497-desc", "enabled": true, "id": "65f22b448e6c4781bcd5d3eb1796e8b9", "name": "tempest-ContainerTest-272815497"}, "audit_ids": ["Mm8FfDIARqmJw6dx0meMew"]}, "serviceCatalog": [{"endpoints": [{"adminURL": "http:\/\/172.16.2.4:8774\/v2.1\/65f22b448e6c4781bcd5d3eb1796e8b9", "region": "regionOne", "internalURL": "http:\/\/172.16.2.4:8774\/v2.1\/65f22b448e6c4781bcd5d3eb1796e8b9", "id": "564951d5f8bd4f26b0049b2738c4ebb5", "publicURL": "http:\/\/10.0.0.101:8774\/v2.1\/65f22b448e6c4781bcd5d3eb1796e8b9"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http:\/\/172.16.2.4:9696\/", "region": "regionOne", "internalURL": "http:\/\/172.16.2.4:9696\/", "id": "05590b4f4975433287c757c2d5c0d4b9", "publicURL": "http:\/\/10.0.0.101:9696\/"}], "endpoints_links": [], "type": "network", "name": "neutron"}, {"endpoints": [{"adminURL": "http:\/\/172.16.2.4:8776\/v2\/65f22b448e6c4781bcd5d3eb1796e8b9", "region": "regionOne", "internalURL": "http:\/\/172.16.2.4:8776\/v2\/65f22b448e6c4781bcd5d3eb1796e8b9", "id": "4f9bdc13e0bc4939a2870b4877d59388", "publicURL": "http:\/\/10.0.0.101:8776\/v2\/65f22b448e6c4781bcd5d3eb1796e8b9"}], "endpoints_links": [], "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL": "http:\/\/172.16.2.4:8042\/", "region": "regionOne", "internalURL": "http:\/\/172.16.2.4:8042\/", "id": "0310c7596c1f45859f5162c58348df71", "publicURL": "http:\/\/10.0.0.101:8042\/"}], "endpoints_links": [], "type": "alarming", "name": "aodh"}, {"endpoints": [{"adminURL": "http:\/\/172.16.2.4:8041\/", "region": "regionOne", "internalURL": "http:\/\/172.16.2.4:8041\/", "id": "042098a261774ce9ac5da1b703877cb9", "publicURL": "http:\/\/10.0.0.101:8041\/"}], "endpoints_links": [], "type": "metric", "name": "gnocchi"}, {"endpoints": [{"adminURL": "http:\/\/172.16.2.4:8777\/", "region": "regionOne", "internalURL": "http:\/\/172.16.2.4:8777\/", "id": "01e56f52214a41efb3e3913e7e7d69d2", "publicURL": "http:\/\/10.0.0.101:8777\/"}], "endpoints_links": [], "type": "metering", "name": "ceilometer"}, {"endpoints": [{"adminURL": "http:\/\/172.16.2.4:8776\/v1\/65f22b448e6c4781bcd5d3eb1796e8b9", "region": "regionOne", "internalURL": "http:\/\/172.16.2.4:8776\/v1\/65f22b448e6c4781bcd5d3eb1796e8b9", "id": "4af0f4336cd045b3ba28581e4101dc81", "publicURL": "http:\/\/10.0.0.101:8776\/v1\/65f22b448e6c4781bcd5d3eb1796e8b9"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http:\/\/172.16.2.4:8004\/v1\/65f22b448e6c4781bcd5d3eb1796e8b9", "region": "regionOne", "internalURL": "http:\/\/172.16.2.4:8004\/v1\/65f22b448e6c4781bcd5d3eb1796e8b9", "id": "0b9f10914c7640288f4520e9d7e6a3ab", "publicURL": "http:\/\/10.0.0.101:8004\/v1\/65f22b448e6c4781bcd5d3eb1796e8b9"}], "endpoints_links": [], "type": "orchestration", "name": "heat"}, {"endpoints": [{"adminURL": "http:\/\/172.16.1.4:8080\/v1", "region": "regionOne", "internalURL": "http:\/\/172.16.1.4:8080\/v1\/AUTH_65f22b448e6c4781bcd5d3eb1796e8b9", "id": "02eab80fd77c45078559cd02a66bb60c", "publicURL": "http:\/\/10.0.0.101:8080\/v1\/AUTH_65f22b448e6c4781bcd5d3eb1796e8b9"}], "endpoints_links": [], "type": "object-store", "name": "swift"}, {"endpoints": [{"adminURL": "http:\/\/172.16.2.4:8386\/v1.1\/65f22b448e6c4781bcd5d3eb1796e8b9", "region": "regionOne", "internalURL": "http:\/\/172.16.2.4:8386\/v1.1\/65f22b448e6c4781bcd5d3eb1796e8b9", "id": "92e773dde49c49f2adfe9a5993ba42ee", "publicURL": "http:\/\/10.0.0.101:8386\/v1.1\/65f22b448e6c4781bcd5d3eb1796e8b9"}], "endpoints_links": [], "type": "data-processing", "name": "sahara"}, {"endpoints": [{"adminURL": "http:\/\/172.16.1.4:9292\/", "region": "regionOne", "internalURL": "http:\/\/172.16.1.4:9292\/", "id": "45af05b43f5f4ba88c8b378bbbde10b4", "publicURL": "http:\/\/10.0.0.101:9292\/"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http:\/\/192.0.2.6:35357\/v2.0", "region": "regionOne", "internalURL": "http:\/\/172.16.2.4:5000\/v2.0", "id": "4076dec3f8c943baa84ad5671983613e",\r\n2016-06-16 07:15:36,708 15907 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_versioned_container): 201 PUT http:\/\/10.0.0.101:8080\/v1\/AUTH_65f22b448e6c4781bcd5d3eb1796e8b9\/tempest-TestVersionContainer-734473029 0.177s\r\n2016-06-16 07:15:36,709 15907 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': ''}\r\nBody: None\r\nResponse - Headers: {'status': '201', 'content-length': '0', 'content-location': 'http:\/\/10.0.0.101:8080\/v1\/AUTH_65f22b448e6c4781bcd5d3eb1796e8b9\/tempest-TestVersionContainer-734473029', 'x-trans-id': 'tx39e5ffb952fc42909a818-0057628a58', 'date': 'Thu, 16 Jun 2016 11:15:36 GMT', 'content-type': 'text\/html; charset=UTF-8', 'connection': 'close'}\r\nBody:\r\n2016-06-16 07:15:36,718 15907 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_versioned_container): 204 HEAD http:\/\/10.0.0.101:8080\/v1\/AUTH_65f22b448e6c4781bcd5d3eb1796e8b9\/tempest-TestVersionContainer-734473029 0.009s\r\n2016-06-16 07:15:36,719 15907 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application\/json', 'Accept': 'application\/json', 'X-Auth-Token': ''}\r\nBody: None\r\nResponse - Headers: {'status': '204', 'content-length': '0', 'x-container-object-count': '0', 'content-location': 'http:\/\/10.0.0.101:8080\/v1\/AUTH_65f22b448e6c4781bcd5d3eb1796e8b9\/tempest-TestVersionContainer-734473029', 'accept-ranges': 'bytes', 'x-storage-policy': 'Policy-0', 'x-container-bytes-used': '0', 'connection': 'close', 'x-timestamp': '1466075736.69614', 'x-trans-id': 'tx88dca78b4ccd4be0a9eb2-0057628a58', 'date': 'Thu, 16 Jun 2016 11:15:36 GMT', 'content-type': 'application\/json; charset=utf-8'}\r\nBody:\r\n2016-06-16 07:15:36,742 15907 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_versioned_container): 201 PUT http:\/\/10.0.0.101:8080\/v1\/AUTH_65f22b448e6c4781bcd5d3eb1796e8b9\/tempest-TestBaseContainer-179915470 0.023s\r\n2016-06-16 07:15:36,743 15907 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-versions-Location': 'tempest-TestVersionContainer-734473029', 'X-Auth-Token': ''}\r\nBody: None\r\nResponse - Headers: {'status': '201', 'content-length': '0', 'content-location': 'http:\/\/10.0.0.101:8080\/v1\/AUTH_65f22b448e6c4781bcd5d3eb1796e8b9\/tempest-TestBaseContainer-179915470', 'x-trans-id': 'tx320c91518c554a5c958aa-0057628a58', 'date': 'Thu, 16 Jun 2016 11:15:36 GMT', 'content-type': 'text\/html; charset=UTF-8', 'connection': 'close'}\r\nBody:\r\n2016-06-16 07:15:36,753 15907 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_versioned_container): 204 HEAD http:\/\/10.0.0.101:8080\/v1\/AUTH_65f22b448e6c4781bcd5d3eb1796e8b9\/tempest-TestBaseContainer-179915470 0.010s\r\n2016-06-16 07:15:36,753 15907 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application\/json', 'Accept': 'application\/json', 'X-Auth-Token': ''}\r\nBody: None\r\nResponse - Headers: {'status': '204', 'content-length': '0', 'x-container-object-count': '0', 'content-location': 'http:\/\/10.0.0.101:8080\/v1\/AUTH_65f22b448e6c4781bcd5d3eb1796e8b9\/tempest-TestBaseContainer-179915470', 'accept-ranges': 'bytes', 'x-storage-policy': 'Policy-0', 'x-container-bytes-used': '0', 'connection': 'close', 'x-timestamp': '1466075736.72735', 'x-trans-id': 'tx81323faab1994f1c97d98-0057628a58', 'date': 'Thu, 16 Jun 2016 11:15:36 GMT', 'content-type': 'application\/json; charset=utf-8'}\r\nBody:\r\n}}}\r\n\r\nTraceback (most recent call last):\r\nFile "tempest\/api\/object_storage\/test_object_version.py", line 69, in test_versioned_container\r\nvers_container_name)\r\nFile "tempest\/api\/object_storage\/test_object_version.py", line 45, in assertContainer\r\nself.assertEqual(header_value, versioned)\r\nFile "\/home\/stack\/tempest\/.tox\/py27\/lib\/python2.7\/site-packages\/testtools\/testcase.py", line 411, in assertEqual\r\nself.assertThat(observed, matcher, message)\r\nFile "\/home\/stack\/tempest\/.tox\/py27\/lib\/python2.7\/site-packages\/testtools\/testcase.py", line 498, in assertThat\r\nraise mismatch_error\r\ntesttools.matchers._impl.MismatchError: 'Missing Header' != 'tempest-TestVersionContainer-734473029'\r\n\r\nRan 1 test in 1.537s\r\nFAILED (failures=1)\r\n<\/pre>\n
Okay, it seems to be a real bug! See, there’s an assert error!
\nFirst we need to find the most important line. For me, it’s the Missing Header line. And let’s see if we found something on google.
\nUsually I remove the quotation marks and random numbers, since these are generated locally. If you run the same test again, the numbers at the end will change. So I looked for:<\/p>\n
\r\nimpl.MismatchError: Missing Header != tempest-TestVersionContainer\r\n<\/pre>\n
But google didn’t bring me anything useful, so it’s time for debug!<\/p>\n
Installing pudb<\/h3>\n
Pudb is a nice python debugger that gives you a very nice ncurses like interface where you can run your code, line by line, and see where’s the error appears
\nInstall it is pretty straightforward:<\/p>\n
\r\npip install pudb\r\n<\/pre>\n
Debugging with pudb<\/h3>\n
In order to debug, you need to edit the python file that you want to debug, import the pudb, and add a pudb.set_trace() where you want to start to debug, in our case, we want to debug tempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container, so let me explain how it works:
\nYou have test separated by dots, where the last part is the name of the test it self, the penultimate is the class name, and then the file and the others are the path, so in our example, test_versioned_container is the name of the test, ContainerTest is the class where the test is, test_object_version is the file, like test_object_version.py, and tempest.api.object_storage is the path, so this test is in tempest\/api\/object_storage in the test_object_version.py file, see, simple!
\nAnd one more tip: Who is the responsible for object_storage? Swift! So we already know where to look for clues<\/p>\n
So, now we know that we need to edit the tempest\/api\/object_storage\/test_object_version.py, let’s do it, and in the beginning of the file, together with the other imports, import pudb:<\/p>\n
\"\"<\/p>\n
Now let’s find our test test_versioned_container in this same file, and add a pudb.set_trace() in the beginning of the test:<\/p>\n
\"\"<\/p>\n
Well, you see that pudb.set_trace() was inserted in the beginning on test_versioned_container, and also, please notice that we found something that might interest us, this test should only run if object_versioning is enabled in swift.
\nSo, let’s run this test now with our pudb and you will see what’s happen:<\/p>\n
\r\npython -m testtools.run tempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container\r\n<\/pre>\n
The first time you run, pudb will open and show some information, just press enter:<\/p>\n
\"\"<\/p>\n
Next he will ask some settings, usually, I just check to show the number of lines, then press right arrow key and enter:<\/p>\n
\"\"<\/p>\n
Now you will see the main window:<\/p>\n
\"\"<\/p>\n
Now you can debug line by line. For example, in the next window you will see that I press n (next line), n again, and then I press control + x in my keyboard, and typed the name of the vers_container_name:<\/p>\n
\"\"<\/p>\n
If you press control + x again, you’ll be back to your code and you can use n for the next line.
\nHere I was executing line by line, until I reach the exception:<\/p>\n
\"\"<\/p>\n
Now that you know where your code is breaking, we can go more deeply in this method self.assertContainer. I’ll press b to add a breakpoint pressing b and let’s rerun our test again. To exit, just press c (continue) that pudb will exit because of the exception.
\nNext run, it will start in the line containing the pudb.set_trace(), and then you can press c and go straight to the line with the breakpoint, there you can press s (step into), and then next, next so on, until you see where it fails:<\/p>\n
\"\"<\/p>\n
So, we see, that the x-versions-location header wasn’t founded, this means that this feature isn’t enabled for us, but for some reason it’s enabled in the CONF.object_storage_feature_enabled.object_versioning, so, let’s see if it’s enabled in our tempest.conf, and we can see that it’s not (sorry, tools\/config_tempest.py isn’t perfect), so let’s see if this is set as enabled by default:<\/p>\n
\r\ngrep "object_versioning" * -R\r\n<\/pre>\n
\r\napi\/object_storage\/test_object_version.py: not CONF.object_storage_feature_enabled.object_versioning,\r\nconfig.py: cfg.BoolOpt('object_versioning',\r\n<\/pre>\n
So, in tempest\/config.py we see that the default is set to True. So, we have two options: The first one, is create a section [object-storage-feature-enabled] and add object_versioning = False and then this test will be skipped, or we can log in our controller and go into \/etc\/swift\/container-server.conf and enable it, and restart the swift service, and of course, open a bug, because this is an error!<\/p>\n
In your controller:<\/p>\n
\r\nsudo crudini --set \/etc\/swift\/container-server.conf DEFAULT allow_versions true\r\nsudo systemctl restart openstack-swift-container*\r\n<\/pre>\n
And let’s go back to undercloud and see if it will work.<\/p>\n
\r\npython -m testtools.run tempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container\r\n<\/pre>\n
\r\nTests running...\r\n\r\nRan 1 test in 4.140s\r\nOK\r\n<\/pre>\n
And congratulations! One more test debugged!
\nAnd that’s it! If you have comments, doubts, or want to add more information, feel free to contact me.<\/p>\n","protected":false},"excerpt":{"rendered":"
Introduction One of the common tasks when you’re working on TripleO is to debug tempest when you found some failures. This is very important because once you found an error, you need to provide all the information needed when you’re gonna fill a bug, or you can even actually, find the root cause of the … Continue reading “Debugging tempest”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false},"categories":[2],"tags":[17,16,18],"jetpack_featured_media_url":"","jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p8bIYB-2t","jetpack-related-posts":[{"id":119,"url":"https:\/\/arxcruz.net\/index.php\/2015\/02\/05\/new-small-project-stackquery-dashboard\/","url_meta":{"origin":153,"position":0},"title":"New small project: stackquery-dashboard","date":"February 5, 2015","format":false,"excerpt":"I\u2019ve started a new project called stackquery-dashboard. Basically, it\u2019s a web interface to show reports about user contribution in OpenStack, using the\u00a0stackalytics.com\u00a0API and part of the code from launchpadstats\u00a0from\u00a0\u00a0Martina Kollarova. The first reason I start this project is because stackalytics.com only shows reports for each user, and sometimes managers in\u2026","rel":"","context":"In "OpenStack"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":138,"url":"https:\/\/arxcruz.net\/index.php\/2015\/06\/09\/c-cedilha-no-fedora-22\/","url_meta":{"origin":153,"position":1},"title":"C cedilha no Fedora 22","date":"June 9, 2015","format":false,"excerpt":"\u00c9 sempre comum, ap\u00f3s um upgrade termos o problema do \u00e7 aparecer como \u0107, para quem usa teclado padr\u00e3o US. No Fedora 22 eu encontrei essa solu\u00e7\u00e3o, que para mim foi muito mais simples do que ficar editando um monte de arquivo. Lembrando que pode ser feito tamb\u00e9m no modo\u2026","rel":"","context":"In "GNOME"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":127,"url":"https:\/\/arxcruz.net\/index.php\/2015\/06\/13\/writing-tests-in-python-for-beginners-part-1-doctest\/","url_meta":{"origin":153,"position":2},"title":"Writing tests in python for beginners - Part 1 - doctest","date":"June 13, 2015","format":false,"excerpt":"So, I just started an interest on python tests and I would like to share what I learned so far. Let\u2019s skip all that theory about how tests are important for the code, bla bla bla, if you want to learn test, probably you already have your own reason, so,\u2026","rel":"","context":"In "Development"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":148,"url":"https:\/\/arxcruz.net\/index.php\/2017\/01\/08\/testing-http-requests-with-mock-or-httmock\/","url_meta":{"origin":153,"position":3},"title":"Testing http requests with mock or HTTMock?","date":"January 8, 2017","format":false,"excerpt":"So, these days I was working in a project that uses requests to make some http\/https calls: And as usual, I wanted to have my unit tests, and requests is very tricky, because, you don't want to keep making requests to the real server, every time you run your unit\u2026","rel":"","context":"In "English"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/arxcruz.net\/index.php\/wp-json\/wp\/v2\/posts\/153"}],"collection":[{"href":"https:\/\/arxcruz.net\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/arxcruz.net\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/arxcruz.net\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/arxcruz.net\/index.php\/wp-json\/wp\/v2\/comments?post=153"}],"version-history":[{"count":3,"href":"https:\/\/arxcruz.net\/index.php\/wp-json\/wp\/v2\/posts\/153\/revisions"}],"predecessor-version":[{"id":164,"href":"https:\/\/arxcruz.net\/index.php\/wp-json\/wp\/v2\/posts\/153\/revisions\/164"}],"wp:attachment":[{"href":"https:\/\/arxcruz.net\/index.php\/wp-json\/wp\/v2\/media?parent=153"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/arxcruz.net\/index.php\/wp-json\/wp\/v2\/categories?post=153"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/arxcruz.net\/index.php\/wp-json\/wp\/v2\/tags?post=153"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}