Debugging tempest

Introduction

One of the common tasks when you’re working on TripleO is to debug tempest when you found some failures. This is very important because once you found an error, you need to provide all the information needed when you’re gonna fill a bug, or you can even actually, find the root cause of the problem, and fix it yourself!
It’s also important to know when a failure in tempest is actually a bug, when is a miss configuration in some .conf file, timeouts, etc.
In this post I’ll try to dig into all these aspects when you’re debugging tempest. But before, we need to make some steps, like setup our OpenStack environment, download tempest, setup the environment and then hands on in some cases.

Installing TripleO

I’ll not cover here how to install TripleO, you can follow the official documentation here.

Preparing the environment

Once you have TripleO installed, we need to access the undercloud, and from there, we need to setup tempest environment. You can use both tempest upstream, or the tempest RDO package. We will use the last one.

Usually, TripleO install everything with the user stack, and in this directory we can find all the scripts and images used to install our environment.

ssh stack@undercloud

Now, we need to source the credentials that TripleO provide to us:

source /home/stack/overcloudrc.v3

There’s an overcloudrc for v2 api, however, it’s going to be deprecated soon.

You also need to have a public network available. If you don’t have it, the follow script can do that for you:

#!/bin/bash

for i in $(neutron floatingip-list -c id -f value)
do
neutron floatingip-disassociate $i
neutron floatingip-delete $i
done
for i in $(neutron router-list -c id -f value); do neutron router-gateway-clear $i; done
for r in $(neutron router-list -c id -f value); do
for p in $(neutron router-port-list $r -c id -f value); do
neutron router-interface-delete $r port=$p || true
done
done
for i in $(neutron router-list -c id -f value); do neutron router-delete $i; done
for i in $(neutron port-list -c id -f value); do neutron port-delete $i; done
for i in $(neutron net-list -c id -f value); do neutron net-delete $i; done

neutron net-create public --router:external=True \
--provider:network_type flat \
--provider:physical_network datacentre

public_net_id=$(neutron net-show public -f value -c id)

neutron subnet-create --name ext-subnet \
--allocation-pool \
start=192.168.24.100,end=192.168.24.120 \
--disable-dhcp \
--gateway 192.168.24.1 \
public 192.168.24.0/24

Set the start and end ip for your external subnet according your network environment.

Install the openstack-tempest, python-junitxml and python-tempestconf:

sudo yum -y install openstack-tempest python-junitxml python-tempestconf

Initialize tempest in a new directory (you don’t need to mkdir this, tempest will do that for you)

tempest init /home/stack/tempest
cd /home/stack/tempest

The output should be something very straightforward:

2017-09-21 08:01:07.215 1049 INFO tempest [-] Using tempest config file /etc/tempest/tempest.conf

Tempestconf

Tempestconf is a python script created to make it easy to create a tempest.conf file. If you ever worked with tempest, you know it’s pretty hard to have everything setup properly manually, and tempestconf tries to detect as much as it can from which services are running in your environment, what you have enabled, and what you don’t, etc.
Assuming you have sourced the overcloudrc.v3 and you have executed the script to create your public network, you can run the following command:

discover-tempest-config --out etc/tempest.conf \
--network-id $public_net_id \
--deployer-input ~/tempest-deployer-input.conf \
--debug --create \
identity.uri $OS_AUTH_URL \
auth.admin_password $OS_PASSWORD \
auth.admin_username $OS_USERNAME \
auth.use_dynamic_credentials true \
compute-feature-enabled.attach_encrypted_volume False \
network.tenant_network_cidr 192.168.0.0/24 \
compute.build_timeout 500 \
volume-feature-enabled.api_v1 False \
validation.image_ssh_user cirros \
validation.ssh_user cirros \
network.build_timeout 500 \
volume.build_timeout 500 \
orchestration.stack_owner_role heat_stack_owner

As you can see, tempest can get information from other files, for example the tempest-deployer-input.conf that is created by TripleO with some information like the region, volume backend being used, etc.
You can also manually set some configs, following the pattern ‘section.config_name value’. For example, in the section validation we are setting the ssh_user option to cirros. In the tempest.conf it will show something like this:

[validation]
ssh_user = cirros

Once the script finishes you will have a tempest.conf in your /home/stack/tempest/etc/ directory that with something like this:

[DEFAULT]
debug = true
use_stderr = false
log_file = tempest.log

[auth]
tempest_roles = _member_
admin_username = admin
admin_project_name = admin
admin_domain_name = Default
use_dynamic_credentials = true
admin_password = Tudv4RGAhwnJq8UPvgbamUAkY

[compute]
image_ssh_user = cirros
region = regionOne
build_timeout = 500
flavor_ref = d8668b47-5d68-4729-b3f6-5eed67eb50fe
flavor_ref_alt = 313dfd3d-059a-4dc2-a6da-6ebc32cfe546
image_ref = 31edf0e4-eed4-4b09-96cd-1f8c9c78887d
image_ref_alt = 7943a836-218c-4902-bf4a-1a36ca52010c

[identity]
username = demo
tenant_name = demo
password = secrete
alt_username = alt_demo
alt_tenant_name = alt_demo
alt_password = secrete
admin_username = admin
admin_tenant_name = admin
admin_domain_name = Default
disable_ssl_certificate_validation = true
region = regionOne
admin_password = Tudv4RGAhwnJq8UPvgbamUAkY
uri = http://192.168.24.7:5000/v2.0
auth_version = v3
uri_v3 = http://192.168.24.7:5000/v3
admin_tenant_id = a1e4e1ee88804a8cacc0796757a5a0c1

[object-storage]
operator_role = SwiftOperator
region = regionOne

[data-processing]

[orchestration]
stack_owner_role = heat_stack_owner
region = regionOne

[scenario]
img_dir = etc

[oslo_concurrency]
lock_path = /tmp

[volume-feature-enabled]
bootable = true
volume_services = true
api_v1 = False
api_v2 = True
api_v3 = True
api_extensions =

[compute-feature-enabled]
live_migration = false
live_migrate_paused_instances = true
preserve_ports = true
console_output = false
attach_encrypted_volume = False
api_extensions = NMN,OS-DCF,OS-EXT-AZ,OS-EXT-IMG-SIZE,OS-EXT-IPS,OS-EXT-IPS-MAC,OS-EXT-SRV-ATTR,OS-EXT-STS,OS-FLV-DISABLED,OS-FLV-EXT-DATA,OS-SCH-HNT,OS-SRV-USG,os-access-ips,os-admin-actions,os-admin-password,os-agents,os-aggregates,os-assisted-volume-snapshots,os-attach-interfaces,os-availability-zone,os-baremetal-ext-status,os-baremetal-nodes,os-block-device-mapping,os-block-device-mapping-v2-boot,os-cell-capacities,os-cells,os-certificates,os-cloudpipe,os-cloudpipe-update,os-config-drive,os-console-auth-tokens,os-console-output,os-consoles,os-create-backup,os-create-server-ext,os-deferred-delete,os-evacuate,os-extended-evacuate-find-host,os-extended-floating-ips,os-extended-hypervisors,os-extended-networks,os-extended-quotas,os-extended-rescue-with-image,os-extended-services,os-extended-services-delete,os-extended-status,os-extended-volumes,os-fixed-ips,os-flavor-access,os-flavor-extra-specs,os-flavor-manage,os-flavor-rxtx,os-flavor-swap,os-floating-ip-dns,os-floating-ip-pools,os-floating-ips,os-floating-ips-bulk,os-fping,os-hide-server-addresses,os-hosts,os-hypervisor-status,os-hypervisors,os-instance-actions,os-instance_usage_audit_log,os-keypairs,os-lock-server,os-migrate-server,os-migrations,os-multiple-create,os-networks,os-networks-associate,os-pause-server,os-personality,os-preserve-ephemeral-rebuild,os-quota-class-sets,os-quota-sets,os-rescue,os-security-group-default-rules,os-security-groups,os-server-diagnostics,os-server-external-events,os-server-group-quotas,os-server-groups,os-server-list-multi-status,os-server-password,os-server-sort-keys,os-server-start-stop,os-services,os-shelve,os-simple-tenant-usage,os-suspend-server,os-tenant-networks,os-used-limits,os-used-limits-for-admin,os-user-data,os-user-quotas,os-virtual-interfaces,os-volume-attachment-update,os-volumes

[network-feature-enabled]
ipv6_subnet_attributes = true
api_extensions = default-subnetpools,qos,network-ip-availability,network_availability_zone,auto-allocated-topology,ext-gw-mode,binding,agent,subnet_allocation,l3_agent_scheduler,tag,external-net,standard-attr-tag,flavors,net-mtu,availability_zone,qos-default,quotas,revision-if-match,l3-ha,provider,multi-provider,quota_details,address-scope,trunk,extraroute,net-mtu-writable,subnet-service-types,standard-attr-timestamp,service-type,qos-rule-type-details,l3-flavors,port-security,extra_dhcp_opt,standard-attr-revisions,pagination,sorting,security-group,dhcp_agent_scheduler,router_availability_zone,rbac-policies,qos-bw-limit-direction,tag-ext,standard-attr-description,router,allowed-address-pairs,project-id,trunk-details

[validation]
image_ssh_user = cirros
ssh_user = cirros

[image]
region = regionOne

[network]
region = regionOne
tenant_network_cidr = 192.168.0.0/24
build_timeout = 500
public_network_id = 9c5f91dd-118e-4637-a3cf-809f672a3c38
floating_network_name = public

[volume]
backend1_name = tripleo_iscsi
region = regionOne
build_timeout = 500

[service_available]
glance = True
manila = False
cinder = True
swift = True
sahara = False
panko = True
nova = True
neutron = True
trove = False
ceilometer = True
ironic = False
heat = True
zaqar = True
gnocchi = True
aodh = True
aodh_plugin = True
horizon = False

[image-feature-enabled]
api_v1 = False
api_v2 = True

[identity-feature-enabled]
api_v2 = True
api_v3 = True
api_extensions = https:,OS-REVOKE,OS-FEDERATION,OS-KSCRUD,OS-SIMPLE-CERT,OS-OAUTH1,OS-EC2
forbid_global_implied_dsr = True

[object-storage-feature-enabled]
discoverable_apis = formpost,bulk_delete,versioned_writes,container_quotas,tempurl,bulk_upload,ratelimit,slo,account_quotas,staticweb

[dashboard]
dashboard_url = http://192.168.24.7/dashboard/
login_url = http://192.168.24.7/dashboard/auth/login/

Testing if everything is working properly

Once we have finished setup tempest, let’s see if everything works properly, you can check if we have some plugins for tempest installed:

tempest list-plugins

+------------------+---------------------------------------------------------+
| Name | EntryPoint |
+------------------+---------------------------------------------------------+
| barbican_tests | barbican_tempest_plugin.plugin:BarbicanTempestPlugin |
| zaqar_tests | zaqar.tests.tempest_plugin.plugin:ZaqarTempestPlugin |
| ceilometer_tests | ceilometer.tests.tempest.plugin:CeilometerTempestPlugin |
| gnocchi_tests | gnocchi.tempest.plugin:GnocchiTempestPlugin |
| aodh_tests | aodh.tests.tempest.plugin:AodhTempestPlugin |
| aws_tests | ec2api.tests.functional.plugin:AWSTempestPlugin |
+------------------+---------------------------------------------------------+

And also list the tests:

ostestr -l
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
${PYTHON:-python} -m subunit.run discover -t /usr/lib/python2.7/site-packages/tempest /usr/lib/python2.7/site-packages/tempest/test_discover --list
tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_create_agent[id-1fc6bdc8-0b6d-4cc7-9f30-9b04fabe5b90]
tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_delete_agent[id-470e0b89-386f-407b-91fd-819737d0b335]
tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents[id-6a326c69-654b-438a-80a3-34bcc454e138]
tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_list_agents_with_filter[id-eabadde4-3cd7-4ec4-a4b5-5a936d2d4408]
tempest.api.compute.admin.test_agents.AgentsAdminTestJSON.test_update_agent[id-dc9ffd51-1c50-4f0e-a820-ae6d2a568a9e]
tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_create_server_with_az[id-96be03c7-570d-409c-90f8-e4db3c646996]
tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_get_details[id-eeef473c-7c52-494d-9f09-2ed7fc8fc036]
tempest.api.compute.admin.test_aggregates.AggregatesAdminTestJSON.test_aggregate_add_host_list[id-7f6a1cc5-2446-4cdb-9baa-b6ae0a919b72]
...

It’s a huge list!

Running tempest

Well, in order to have something to debug, we must run some tests, and here we can first save the actual state of the cloud:

tempest cleanup --init-saved-state

This will create a saved_state.json file with the information about how our environment was before we ran tempest. This is good because if some test fails in the tearDownClass step, it might leave some dirty behind, and we can cleanup later.

Smoke

Tempest has a set of tests called smoke, these tests are supposed to be fast and can verify the sanity of your system, and that’s what we going to run here:

ostestr smoke

Debugging tempest

And now what everybody was expecting!
Now that we ran tempest and we have our results, we can start to debug these errors.

Type of errors

In my opinion, tempest have several types of errors, there are some errors that it’s actually a bug, and you need to report, others are errors due miss-configuration files, others because some timeout we will try to cover mostly of the types here

Getting the list of errors

When tempest finishes to run, it shows your a report with the number of tests executed, the success and failures, like this one:

======
Totals
======
Ran: 1522 tests in 3417.0000 sec.
- Passed: 1408
- Skipped: 87
- Expected Fail: 0
- Unexpected Success: 0
- Failed: 27
Sum of execute time for each test: 7235.4187 sec.

==============
Worker Balance
==============
- Worker 0 (367 tests) => 0:45:38.887316
- Worker 1 (384 tests) => 0:38:19.733741
- Worker 2 (325 tests) => 0:56:47.206368
- Worker 3 (446 tests) => 0:46:33.747285

But that’s not enough to us, we need to get the list of errors, for that, we can use the testr tool to get the failing errors, for example, when you execute the following command:

testr failing |grep FAIL | sed -e 's/\[.*\].*//g' -e 's/FAIL\: //'

We will see something like this:

tempest.api.compute.certificates.test_certificates.CertificatesV2TestJSON.test_create_root_certificate
tempest.api.compute.certificates.test_certificates.CertificatesV2TestJSON.test_get_root_certificate
setUpClass (tempest.api.orchestration.stacks.test_nova_keypair_resources.NovaKeyPairResourcesYAMLTest)
setUpClass (tempest.api.orchestration.stacks.test_swift_resources.SwiftResourcesTestJSON)
tempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container
setUpClass (tempest.api.orchestration.stacks.test_templates.TemplateYAMLTestJSON)
setUpClass (tempest.api.orchestration.stacks.test_templates_negative.TemplateAWSNegativeTestJSON)
setUpClass (tempest.api.orchestration.stacks.test_volumes.CinderResourcesTest)
setUpClass (tempest.api.orchestration.stacks.test_limits.TestServerStackLimits)
setUpClass (tempest.api.orchestration.stacks.test_non_empty_stack.StacksTestJSON)
setUpClass (tempest.api.orchestration.stacks.test_resource_types.ResourceTypesTest)
setUpClass (tempest.api.orchestration.stacks.test_soft_conf.TestSoftwareConfig)
setUpClass (tempest.api.orchestration.stacks.test_templates_negative.TemplateYAMLNegativeTestJSON)
tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV1Test.test_volume_backup_create_get_detailed_list_restore_delete
tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV1Test.test_volume_backup_export_import
tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_cryptsetup
tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV2Test.test_volume_backup_create_get_detailed_list_restore_delete
tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV2Test.test_volume_backup_export_import
setUpClass (tempest.api.orchestration.stacks.test_environment.StackEnvironmentTest)
setUpClass (tempest.api.orchestration.stacks.test_nova_keypair_resources.NovaKeyPairResourcesAWSTest)
setUpClass (tempest.api.orchestration.stacks.test_stacks.StacksTestJSON)
setUpClass (tempest.api.orchestration.stacks.test_templates.TemplateAWSTestJSON)
tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_luks
tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_reboot
tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario
tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic
tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_multiple_security_groups

setUpClass

You have noticed that some failures start with this setUpClass, this means, that the test haven’t even started because some error were found when the test was setting whatever the test do (create VM’s, networks, etc).

tempest.something.something_else.MyClassV2.test_that_fails

When you see some failure like that, it means that the test pass from the setUpClass phase, and actually tried to run, and it fails for some reason, don’t worry, we will debug it.

tearDownClass

Some tests fails when it tries to clean the test, for example, in the setUpClass phase, it creates a VM, but in the tearDownClass the VM wasn’t able to be deleted, it’s rare, but it can happen.

The debug!

Let’s try to debug something.

Re-running one particular test

The easiest way to do that is using python testtools with the following command:

python -m testtools.run tempest.test.that.i.want.to.run

Why not use ostestr to run that particular test? Because, we are going to use a tool called pudb, and you can’t use it with ostestr.

Was it a timeout?

The first thing we do when we are debugging, is to re-run the test again to be sure that wasn’t a failure based on some timeout, sometimes the machine or network are struggling and we have a timeout. Tempest have a timeout that is configured in etc/tempest.conf that can, for example, tell that if a VM doesn’t reach the status ACTIVE in 60 seconds, mark the test as a failure, but if you’re running tests in parallel, the machine might take more time to reach ACTIVE because all resources are full for example. So, it’s always a good idea re-run the test, and also, rerunning the test will show the output of the failure, which is a good source of information.

So, let’s see a few examples of failures and how we debug it:

Let’s get the first failure test tempest.api.compute.certificates.test_certificates.CertificatesV2TestJSON.test_create_root_certificate

python -m testtools.run tempest.api.compute.certificates.test_certificates.CertificatesV2TestJSON.test_create_root_certificate

And after a while, here’s the output:

Tests running...
======================================================================
FAIL: tempest.api.compute.certificates.test_certificates.CertificatesV2TestJSON.test_create_root_certificate[id-c070a441-b08e-447e-a733-905909535b1b]
----------------------------------------------------------------------
pythonlogging:'': {{{
2016-06-15 10:18:15,733 28198 INFO [tempest.lib.common.rest_client] Request (CertificatesV2TestJSON:test_create_root_certificate): 504 POST http://10.0.0.101:8774/v2.1/832848e6175148aa874dbb2a891cd02a/os-certificates 60.004s
2016-06-15 10:18:15,734 28198 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': ''}
Body: None
Response - Headers: {'status': '504', 'connection': 'close', 'content-type': 'text/html', 'content-location': 'http://10.0.0.101:8774/v2.1/832848e6175148aa874dbb2a891cd02a/os-certificates', 'cache-control': 'no-cache'}
Body:

<h1>504 Gateway Time-out</h1>

The server didn't respond in time.

}}}

Traceback (most recent call last):
File "tempest/api/compute/certificates/test_certificates.py", line 34, in test_create_root_certificate
body = self.certificates_client.create_certificate()['certificate']
File "tempest/lib/services/compute/certificates_client.py", line 35, in create_certificate
resp, body = self.post(url, None)
File "tempest/lib/common/rest_client.py", line 270, in post
return self.request('POST', url, extra_headers, headers, body, chunked)
File "tempest/lib/services/compute/base_compute_client.py", line 53, in request
method, url, extra_headers, headers, body, chunked)
File "tempest/lib/common/rest_client.py", line 664, in request
resp, resp_body)
File "tempest/lib/common/rest_client.py", line 832, in _error_checker
resp=resp)
tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received
Details: 504

Ran 1 test in 61.744s
FAILED (failures=1)

Notice the lines 9-16. The first thing that we can notice is that we got a Gateway Time-out.
Checking the name of the test, it’s trying to create a root certificate, and check the content-location: http://10.0.0.101:8774/v2.1/832848e6175148aa874dbb2a891cd02a/os-certificates
It’s hard to figure out, but in this case, os-certificates it’s from a service, openstack-nova-cert, so, the first thing that we need to check is if this service is up and running in our controller, so, we need to ssh to the controller machine and check it.
You already know the IP address that is 10.0.0.101, and the user is heat-admin

ssh heat-admin@10.0.0.101

In order to ssh works, you need to be logged as the ‘stack’ user

Now in the controller machine, let’s see the status of the services:

systemctl status openstack-*

This will show a list of all services, and you can notice this:

● openstack-nova-cert.service - OpenStack Nova Cert Server
Loaded: loaded (/usr/lib/systemd/system/openstack-nova-cert.service; enabled; vendor preset: disabled)

As you can see, the openstack-nova-cert service is INACTIVE!!!
Let’s start it

sudo systemctl start openstack-nova-cert

And check if the service was successfuly sarted:

systemctl status openstack-nova-cert
systemctl status openstack-nova-cert

And here’s the output:

● openstack-nova-cert.service - OpenStack Nova Cert Server
Loaded: loaded (/usr/lib/systemd/system/openstack-nova-cert.service; disabled; vendor preset: disabled)
Active: active (running) since Qua 2017-09-21 14:46:43 UTC; 51s ago
Main PID: 6985 (nova-cert)
CGroup: /system.slice/openstack-nova-cert.service
└─6985 /usr/bin/python2 /usr/bin/nova-cert

Now, let’s go back to our undercloud, and try again run the test. Please remember to activate the virtualenv again.

python -m testtools.run tempest.api.compute.certificates.test_certificates.CertificatesV2TestJSON.test_create_root_certificate

And now we have a different output:

Tests running...

Ran 1 test in 1.153s
OK

And congratulations! You have debugged your first test in tempest!
But this was a easy one, let’s go for something more challenging

More challenge stuff

Let’s get the next failure tempest.api.orchestration.stacks.test_nova_keypair_resources.NovaKeyPairResourcesYAMLTest

python -m testtools.run tempest.api.orchestration.stacks.test_nova_keypair_resources.NovaKeyPairResourcesYAMLTest

Output:

Tests running...
======================================================================
ERROR: setUpClass (tempest.api.orchestration.stacks.test_nova_keypair_resources.NovaKeyPairResourcesYAMLTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tempest/test.py", line 278, in setUpClass
six.reraise(etype, value, trace)
File "tempest/test.py", line 266, in setUpClass
cls.setup_credentials()
File "tempest/api/orchestration/base.py", line 40, in setup_credentials
cls.os = cls.get_client_manager(roles=[stack_owner_role])
File "tempest/test.py", line 536, in get_client_manager
creds = cred_provider.get_creds_by_roles(**params)
File "tempest/common/dynamic_creds.py", line 323, in get_creds_by_roles
return self.get_credentials(roles)
File "tempest/common/dynamic_creds.py", line 286, in get_credentials
credentials = self._create_creds(roles=credential_type)
File "tempest/common/dynamic_creds.py", line 157, in _create_creds
self.creds_client.assign_user_role(user, project, role)
File "tempest/common/cred_client.py", line 72, in assign_user_role
raise lib_exc.NotFound(msg)
tempest.lib.exceptions.NotFound: Object not found
Details: No "heat_stack_owner" role found

Ran 0 tests in 4.887s
FAILED (failures=1)

Now, things are getting interested here, it wasn’t a timeout, it wasn’t some service that weren’t enabled, but you can see in the details that was a role that wasn’t founded.
so, let’s verify first which roles we have in keystone:

source /home/stack/overcloudrc.v3
openstack role list
+----------------------------------+------------------+
| ID | Name |
+----------------------------------+------------------+
| 3d7ddc928c3044988b204730a3087d0d | swiftoperator |
| 7608609651af49369aa68510715a3507 | admin |
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_ |
| cc2203d86a2a496c9b62876bbe8ffaa0 | heat_stack_user |
| cfd9df98de374b678f756d1ed8dd3662 | ResellerAdmin |
+----------------------------------+------------------+

So, as you can see, the role is heat_stack_user, instead of heat_stack_owner, but why tempest is trying to use heat_stack_owner?
Let’s see:

cd /usr/lib/python2.7/site-packages/tempest
grep heat_stack_owner * -R
config.py: cfg.StrOpt('stack_owner_role', default='heat_stack_owner',

Now, that’s interesting, if you notice tempest/config.py, the default role is heat_stack_owner.
Tempest have default values for everything that you can set in etc/tempest.conf. If there’s nothing set in tempest.conf file, tempest will use the default one, so we have here two options: creat
e a heat_stack_owner role, or change/add the stack_owner_role option in our etc/tempest.conf. I believe the second option is better so let’s do it.
How do we know which section we need to add it?
If you open the tempest/config.py and look for the heat_stack_owner, you will see that it’s under OrchestrationGroup:

orchestration_group = cfg.OptGroup(name='orchestration',
title='Orchestration Service Options')

OrchestrationGroup = [
cfg.StrOpt('catalog_type',
default='orchestration',
help="Catalog type of the Orchestration service."),
cfg.StrOpt('region',
default='',
help="The orchestration region name to use. If empty, the "
"value of identity.region is used instead. If no such "
"region is found in the service catalog, the first found "
"one is used."),
cfg.StrOpt('endpoint_type',
default='publicURL',
choices=['public', 'admin', 'internal',
'publicURL', 'adminURL', 'internalURL'],
help="The endpoint type to use for the orchestration service."),
cfg.StrOpt('stack_owner_role', default='heat_stack_owner',
...
(orchestration_group, OrchestrationGroup),

OrchestrationGroup became orechestration in etc/tempest.conf. So, let’s go there and add the following line under [orchestration] in etc/tempest.conf

stack_owner_role = heat_stack_user

Let’s try again?

python -m testtools.run tempest.api.orchestration.stacks.test_nova_keypair_resources.NovaKeyPairResourcesYAMLTest

And the output:

Tests running...
======================================================================
ERROR: setUpClass (tempest.api.orchestration.stacks.test_nova_keypair_resources.NovaKeyPairResourcesYAMLTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tempest/test.py", line 278, in setUpClass
six.reraise(etype, value, trace)
File "tempest/test.py", line 271, in setUpClass
cls.resource_setup()
File "tempest/api/orchestration/stacks/test_nova_keypair_resources.py", line 35, in resource_setup
'KeyPairName2': cls.stack_name + '_2'
File "tempest/api/orchestration/base.py", line 77, in create_stack
files=files)
File "tempest/services/orchestration/json/orchestration_client.py", line 56, in create_stack
resp, body = self.post(uri, headers=headers, body=body)
File "tempest/lib/common/rest_client.py", line 270, in post
return self.request('POST', url, extra_headers, headers, body, chunked)
File "tempest/lib/common/rest_client.py", line 664, in request
resp, resp_body)
File "tempest/lib/common/rest_client.py", line 757, in _error_checker
raise exceptions.Forbidden(resp_body, resp=resp)
tempest.lib.exceptions.Forbidden: Forbidden
Details: {u'title': u'Forbidden', u'explanation': u'Access was denied to this resource.', u'code': 403, u'error': {u'message': u'You are not authorized to use create.', u'traceback': None, u'type': u'Forbidden'}}

Ran 0 tests in 9.718s
FAILED (failures=1)

And now we have a different failure, in this case, the role doesn’t have the rights to add these resources.
Well, we can then use _member_ role, since this was the default role for everything in tempest before they start to create roles separately. Let’s change again our etc/tempest.conf in [orchestration] section:

stack_owner_role = _member_

Running again…

python -m testtools.run tempest.api.orchestration.stacks.test_nova_keypair_resources.NovaKeyPairResourcesYAMLTest

And the output:

Tests running...

Ran 2 tests in 16.784s
OK

The pudb in action

Here is where things get interesting, let’s check the next failure in our list:

python -m testtools.run tempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container
Tests running...

======================================================================
FAIL: tempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container[id-a151e158-dcbf-4a1f-a1e7-46cd65895a6f]
----------------------------------------------------------------------
pythonlogging:'': {{{
2016-06-16 07:15:36,531 15907 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_versioned_container): 200 POST http://10.0.0.101:5000/v2.0/tokens
2016-06-16 07:15:36,531 15907 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json'}
Body:
Response - Headers: {'status': '200', 'content-length': '4540', 'content-location': 'http://10.0.0.101:5000/v2.0/tokens', 'vary': 'X-Auth-Token,Accept-Encoding', 'server': 'Apache/2.4.6 (Red Hat Enterprise Linux)', 'connection': 'close', 'date': 'Thu, 16 Jun 2016 11:15:36 GMT', 'content-type': 'application/json', 'x-openstack-request-id': 'req-f79820a7-d01a-4ac2-a6ea-ac76f63f6070'}
Body: {"access": {"token": {"issued_at": "2016-06-16T11:15:36.517581Z", "expires": "2016-06-16T12:15:36Z", "id": "28fba048ed524fdcb5309a6cd2e6975c", "tenant": {"description": "tempest-ContainerTest-272815497-desc", "enabled": true, "id": "65f22b448e6c4781bcd5d3eb1796e8b9", "name": "tempest-ContainerTest-272815497"}, "audit_ids": ["Mm8FfDIARqmJw6dx0meMew"]}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://172.16.2.4:8774/v2.1/65f22b448e6c4781bcd5d3eb1796e8b9", "region": "regionOne", "internalURL": "http://172.16.2.4:8774/v2.1/65f22b448e6c4781bcd5d3eb1796e8b9", "id": "564951d5f8bd4f26b0049b2738c4ebb5", "publicURL": "http://10.0.0.101:8774/v2.1/65f22b448e6c4781bcd5d3eb1796e8b9"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://172.16.2.4:9696/", "region": "regionOne", "internalURL": "http://172.16.2.4:9696/", "id": "05590b4f4975433287c757c2d5c0d4b9", "publicURL": "http://10.0.0.101:9696/"}], "endpoints_links": [], "type": "network", "name": "neutron"}, {"endpoints": [{"adminURL": "http://172.16.2.4:8776/v2/65f22b448e6c4781bcd5d3eb1796e8b9", "region": "regionOne", "internalURL": "http://172.16.2.4:8776/v2/65f22b448e6c4781bcd5d3eb1796e8b9", "id": "4f9bdc13e0bc4939a2870b4877d59388", "publicURL": "http://10.0.0.101:8776/v2/65f22b448e6c4781bcd5d3eb1796e8b9"}], "endpoints_links": [], "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"adminURL": "http://172.16.2.4:8042/", "region": "regionOne", "internalURL": "http://172.16.2.4:8042/", "id": "0310c7596c1f45859f5162c58348df71", "publicURL": "http://10.0.0.101:8042/"}], "endpoints_links": [], "type": "alarming", "name": "aodh"}, {"endpoints": [{"adminURL": "http://172.16.2.4:8041/", "region": "regionOne", "internalURL": "http://172.16.2.4:8041/", "id": "042098a261774ce9ac5da1b703877cb9", "publicURL": "http://10.0.0.101:8041/"}], "endpoints_links": [], "type": "metric", "name": "gnocchi"}, {"endpoints": [{"adminURL": "http://172.16.2.4:8777/", "region": "regionOne", "internalURL": "http://172.16.2.4:8777/", "id": "01e56f52214a41efb3e3913e7e7d69d2", "publicURL": "http://10.0.0.101:8777/"}], "endpoints_links": [], "type": "metering", "name": "ceilometer"}, {"endpoints": [{"adminURL": "http://172.16.2.4:8776/v1/65f22b448e6c4781bcd5d3eb1796e8b9", "region": "regionOne", "internalURL": "http://172.16.2.4:8776/v1/65f22b448e6c4781bcd5d3eb1796e8b9", "id": "4af0f4336cd045b3ba28581e4101dc81", "publicURL": "http://10.0.0.101:8776/v1/65f22b448e6c4781bcd5d3eb1796e8b9"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://172.16.2.4:8004/v1/65f22b448e6c4781bcd5d3eb1796e8b9", "region": "regionOne", "internalURL": "http://172.16.2.4:8004/v1/65f22b448e6c4781bcd5d3eb1796e8b9", "id": "0b9f10914c7640288f4520e9d7e6a3ab", "publicURL": "http://10.0.0.101:8004/v1/65f22b448e6c4781bcd5d3eb1796e8b9"}], "endpoints_links": [], "type": "orchestration", "name": "heat"}, {"endpoints": [{"adminURL": "http://172.16.1.4:8080/v1", "region": "regionOne", "internalURL": "http://172.16.1.4:8080/v1/AUTH_65f22b448e6c4781bcd5d3eb1796e8b9", "id": "02eab80fd77c45078559cd02a66bb60c", "publicURL": "http://10.0.0.101:8080/v1/AUTH_65f22b448e6c4781bcd5d3eb1796e8b9"}], "endpoints_links": [], "type": "object-store", "name": "swift"}, {"endpoints": [{"adminURL": "http://172.16.2.4:8386/v1.1/65f22b448e6c4781bcd5d3eb1796e8b9", "region": "regionOne", "internalURL": "http://172.16.2.4:8386/v1.1/65f22b448e6c4781bcd5d3eb1796e8b9", "id": "92e773dde49c49f2adfe9a5993ba42ee", "publicURL": "http://10.0.0.101:8386/v1.1/65f22b448e6c4781bcd5d3eb1796e8b9"}], "endpoints_links": [], "type": "data-processing", "name": "sahara"}, {"endpoints": [{"adminURL": "http://172.16.1.4:9292/", "region": "regionOne", "internalURL": "http://172.16.1.4:9292/", "id": "45af05b43f5f4ba88c8b378bbbde10b4", "publicURL": "http://10.0.0.101:9292/"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://192.0.2.6:35357/v2.0", "region": "regionOne", "internalURL": "http://172.16.2.4:5000/v2.0", "id": "4076dec3f8c943baa84ad5671983613e",
2016-06-16 07:15:36,708 15907 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_versioned_container): 201 PUT http://10.0.0.101:8080/v1/AUTH_65f22b448e6c4781bcd5d3eb1796e8b9/tempest-TestVersionContainer-734473029 0.177s
2016-06-16 07:15:36,709 15907 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-Auth-Token': ''}
Body: None
Response - Headers: {'status': '201', 'content-length': '0', 'content-location': 'http://10.0.0.101:8080/v1/AUTH_65f22b448e6c4781bcd5d3eb1796e8b9/tempest-TestVersionContainer-734473029', 'x-trans-id': 'tx39e5ffb952fc42909a818-0057628a58', 'date': 'Thu, 16 Jun 2016 11:15:36 GMT', 'content-type': 'text/html; charset=UTF-8', 'connection': 'close'}
Body:
2016-06-16 07:15:36,718 15907 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_versioned_container): 204 HEAD http://10.0.0.101:8080/v1/AUTH_65f22b448e6c4781bcd5d3eb1796e8b9/tempest-TestVersionContainer-734473029 0.009s
2016-06-16 07:15:36,719 15907 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': ''}
Body: None
Response - Headers: {'status': '204', 'content-length': '0', 'x-container-object-count': '0', 'content-location': 'http://10.0.0.101:8080/v1/AUTH_65f22b448e6c4781bcd5d3eb1796e8b9/tempest-TestVersionContainer-734473029', 'accept-ranges': 'bytes', 'x-storage-policy': 'Policy-0', 'x-container-bytes-used': '0', 'connection': 'close', 'x-timestamp': '1466075736.69614', 'x-trans-id': 'tx88dca78b4ccd4be0a9eb2-0057628a58', 'date': 'Thu, 16 Jun 2016 11:15:36 GMT', 'content-type': 'application/json; charset=utf-8'}
Body:
2016-06-16 07:15:36,742 15907 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_versioned_container): 201 PUT http://10.0.0.101:8080/v1/AUTH_65f22b448e6c4781bcd5d3eb1796e8b9/tempest-TestBaseContainer-179915470 0.023s
2016-06-16 07:15:36,743 15907 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'X-versions-Location': 'tempest-TestVersionContainer-734473029', 'X-Auth-Token': ''}
Body: None
Response - Headers: {'status': '201', 'content-length': '0', 'content-location': 'http://10.0.0.101:8080/v1/AUTH_65f22b448e6c4781bcd5d3eb1796e8b9/tempest-TestBaseContainer-179915470', 'x-trans-id': 'tx320c91518c554a5c958aa-0057628a58', 'date': 'Thu, 16 Jun 2016 11:15:36 GMT', 'content-type': 'text/html; charset=UTF-8', 'connection': 'close'}
Body:
2016-06-16 07:15:36,753 15907 INFO [tempest.lib.common.rest_client] Request (ContainerTest:test_versioned_container): 204 HEAD http://10.0.0.101:8080/v1/AUTH_65f22b448e6c4781bcd5d3eb1796e8b9/tempest-TestBaseContainer-179915470 0.010s
2016-06-16 07:15:36,753 15907 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': ''}
Body: None
Response - Headers: {'status': '204', 'content-length': '0', 'x-container-object-count': '0', 'content-location': 'http://10.0.0.101:8080/v1/AUTH_65f22b448e6c4781bcd5d3eb1796e8b9/tempest-TestBaseContainer-179915470', 'accept-ranges': 'bytes', 'x-storage-policy': 'Policy-0', 'x-container-bytes-used': '0', 'connection': 'close', 'x-timestamp': '1466075736.72735', 'x-trans-id': 'tx81323faab1994f1c97d98-0057628a58', 'date': 'Thu, 16 Jun 2016 11:15:36 GMT', 'content-type': 'application/json; charset=utf-8'}
Body:
}}}

Traceback (most recent call last):
File "tempest/api/object_storage/test_object_version.py", line 69, in test_versioned_container
vers_container_name)
File "tempest/api/object_storage/test_object_version.py", line 45, in assertContainer
self.assertEqual(header_value, versioned)
File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
self.assertThat(observed, matcher, message)
File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
raise mismatch_error
testtools.matchers._impl.MismatchError: 'Missing Header' != 'tempest-TestVersionContainer-734473029'

Ran 1 test in 1.537s
FAILED (failures=1)

Okay, it seems to be a real bug! See, there’s an assert error!
First we need to find the most important line. For me, it’s the Missing Header line. And let’s see if we found something on google.
Usually I remove the quotation marks and random numbers, since these are generated locally. If you run the same test again, the numbers at the end will change. So I looked for:

impl.MismatchError: Missing Header != tempest-TestVersionContainer

But google didn’t bring me anything useful, so it’s time for debug!

Installing pudb

Pudb is a nice python debugger that gives you a very nice ncurses like interface where you can run your code, line by line, and see where’s the error appears
Install it is pretty straightforward:

pip install pudb

Debugging with pudb

In order to debug, you need to edit the python file that you want to debug, import the pudb, and add a pudb.set_trace() where you want to start to debug, in our case, we want to debug tempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container, so let me explain how it works:
You have test separated by dots, where the last part is the name of the test it self, the penultimate is the class name, and then the file and the others are the path, so in our example, test_versioned_container is the name of the test, ContainerTest is the class where the test is, test_object_version is the file, like test_object_version.py, and tempest.api.object_storage is the path, so this test is in tempest/api/object_storage in the test_object_version.py file, see, simple!
And one more tip: Who is the responsible for object_storage? Swift! So we already know where to look for clues

So, now we know that we need to edit the tempest/api/object_storage/test_object_version.py, let’s do it, and in the beginning of the file, together with the other imports, import pudb:

Now let’s find our test test_versioned_container in this same file, and add a pudb.set_trace() in the beginning of the test:

Well, you see that pudb.set_trace() was inserted in the beginning on test_versioned_container, and also, please notice that we found something that might interest us, this test should only run if object_versioning is enabled in swift.
So, let’s run this test now with our pudb and you will see what’s happen:

python -m testtools.run tempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container

The first time you run, pudb will open and show some information, just press enter:

Next he will ask some settings, usually, I just check to show the number of lines, then press right arrow key and enter:

Now you will see the main window:

Now you can debug line by line. For example, in the next window you will see that I press n (next line), n again, and then I press control + x in my keyboard, and typed the name of the vers_container_name:

If you press control + x again, you’ll be back to your code and you can use n for the next line.
Here I was executing line by line, until I reach the exception:

Now that you know where your code is breaking, we can go more deeply in this method self.assertContainer. I’ll press b to add a breakpoint pressing b and let’s rerun our test again. To exit, just press c (continue) that pudb will exit because of the exception.
Next run, it will start in the line containing the pudb.set_trace(), and then you can press c and go straight to the line with the breakpoint, there you can press s (step into), and then next, next so on, until you see where it fails:

So, we see, that the x-versions-location header wasn’t founded, this means that this feature isn’t enabled for us, but for some reason it’s enabled in the CONF.object_storage_feature_enabled.object_versioning, so, let’s see if it’s enabled in our tempest.conf, and we can see that it’s not (sorry, tools/config_tempest.py isn’t perfect), so let’s see if this is set as enabled by default:

grep "object_versioning" * -R
api/object_storage/test_object_version.py: not CONF.object_storage_feature_enabled.object_versioning,
config.py: cfg.BoolOpt('object_versioning',

So, in tempest/config.py we see that the default is set to True. So, we have two options: The first one, is create a section [object-storage-feature-enabled] and add object_versioning = False and then this test will be skipped, or we can log in our controller and go into /etc/swift/container-server.conf and enable it, and restart the swift service, and of course, open a bug, because this is an error!

In your controller:

sudo crudini --set /etc/swift/container-server.conf DEFAULT allow_versions true
sudo systemctl restart openstack-swift-container*

And let’s go back to undercloud and see if it will work.

python -m testtools.run tempest.api.object_storage.test_object_version.ContainerTest.test_versioned_container
Tests running...

Ran 1 test in 4.140s
OK

And congratulations! One more test debugged!
And that’s it! If you have comments, doubts, or want to add more information, feel free to contact me.

0 thoughts on “Debugging tempest”

  1. car sale australia
    Произвольный текст и макросы до содержимого файла
    Credit and finance $ Video – -Credit and finance] Credit and finance Bad credit car finance Have you been turned down by other lenders because of your bad credit history? Worried about having poor credit? We consider finance applications even if you’ve been declined elsewhere. Bad credit history If you’ve been suffering with a bad credit history Moneybarn are the bad credit car finance specialist. We understand that you still need a reliable vehicle so that you can get to work, click below to find out how we can help you. Bad credit history If you’ve been suffering with …
    The post Произвольный текст и макросы до содержимого файла appeared first on Auto&Car.

    Massachusetts Business

  2. how do you get life insurance
    10 AUTO ABBANDONATE PIU CARE MAI TROVATE. & Video
    10 AUTO ABBANDONATE PIU CARE MAI TROVATE Convincing someone to be a co-signer may not be easy, c3422 Engine. This is via a 7-day trial — after that, patriots Sign OL Bowanko. 10 AUTO ABBANDONATE PIU CARE MAI TROVATE further notice, satellite Internet Connection. The all inclusive package usually contains the price for the room, eM +39 052 10 AUTO ABBANDONATE PIU CARE MAI TROVATE 2561 Viale Sant Ambrogio 10 AUTO ABBANDONATE PIU CARE MAI TROVATE 33. But remember to look at the interest and charges, credit 10 AUTO ABBANDONATE PIU CARE MAI TROVATE on card. But they do not …
    The post 10 AUTO ABBANDONATE PIU CARE MAI TROVATE. & Video appeared first on Car.

    Vermont Finance
    visterra credit
    taxmasters bronx
    auto
    com
    posb mysavings

    BONUS TOP 250 DOFOLLOW TRUST LINK LIST SITE

    http://4.remmont.com/2019/03/23/do-i-have-to-repay-my-creditors-in-bankruptcy-if-i-own-a-home-hampton-newport-news-video/
    http://6.remmont.com/2019/04/08/divani-chateau-d-ax-catalogo-con-prezzi-cucine-a-angolo-prezzi/
    http://6.remmont.com/2019/04/02/self-drive-hire-insurance-brokers-video-staying-safe-in-the-luxury-and-supercar-hire-industry-premiere-velocity/
    http://8.remmont.com/2019/04/08/cheap-insurance-louisville/
    http://7.remmont.com/2019/04/01/difference-between-for-profit-colleges-and-nonprofit-colleges/
    http://9.remmont.com/2019/04/20/american-flood-insurance-video/
    http://8.remmont.com/2019/03/26/what-deductible-should-i-choose-video/
    http://1.remmont.com/2019/03/pain-free-private-cloud/
    http://1.remmont.com/2019/04/movers-san-diego-video-san-diego-team-two-men-and-a-truck/
    http://3.remmont.com/2019/04/02/victorian-flood-warnings-and-video-victoria-abc-emergency/
    http://10.remmont.com/2019/03/23/weddings-in-houston/
    http://3.remmont.com/2019/03/21/san-diego-beaches-video/
    http://9.remmont.com/2019/04/03/10-best-paying-jobs-that-dont-require-a-college-degree/
    http://8.remmont.com/2019/04/08/cina-ultime-notizie-su-cina-argomenti-del-sole-24-ore-ultime-notizie-economia-e-finanza/
    http://5.remmont.com/2019/04/09/direct-travel-and-video/
    http://4.remmont.com/2019/03/22/slip-n-slide-in-moving-truck-video/
    http://7.remmont.com/2019/03/31/pro-lab-home-safety-test-kits/
    http://6.remmont.com/2019/04/02/short-term-disability-insurance-california-video-basic-and-voluntary-disability-ucnet/
    http://4.remmont.com/2019/03/21/resorts-san-diego-video/
    http://3.remmont.com/2019/03/21/credit-bureaus-are-stealing-and-selling-your-personal-information-and-video/
    http://3.remmont.com/2019/03/24/3-tips-to-get-work-as-a-claims-adjuster-video/
    http://6.remmont.com/2019/04/01/gunstige-lkw-versicherung-video-versicherungsvergleich-lkw/
    http://3.remmont.com/2019/03/31/sacramento-air-conditioning-service/
    http://6.remmont.com/2019/04/02/what-is-dedicated-server-webopedia-definition/
    http://10.remmont.com/2019/03/20/what-apartments-to-buy-cardonezone-video/
    http://7.remmont.com/2019/04/08/versicherung-katze-video/
    http://3.remmont.com/2019/03/26/citizens-property-insurance-urges-customers-to-drop-state-run-coverage-video/
    http://1.remmont.com/2019/03/confronta-prezzi-assicurazioni-video/
    http://9.remmont.com/2019/04/08/book-condor-fluege/
    http://10.remmont.com/2019/04/01/direct-vent-fireplace-systems/
    http://1.remmont.com/2019/04/2016-minnesota-statutes/
    http://3.remmont.com/2019/03/31/free-school-supplies-where-to-find-them/
    http://1.remmont.com/2019/03/assicurazione-casco-video/
    http://3.remmont.com/2019/04/09/florida-auto-insurance-from-nationwide-auto-insurance-in-florida/
    http://2.remmont.com/2019/04/19/contents-insurance-for-rented-property-video/
    http://4.remmont.com/2019/04/09/where-to-go-in-tennessee-and-video/
    http://4.remmont.com/2019/04/08/michigan-colleges-online/
    http://5.remmont.com/2019/04/11/il-ducato-arredamento-bagno-e-mobili-bagno-produzione-di-mobili-bagno-e-arredamento-bagno-a-pesaro-idee-bagno-moderno/
    http://6.remmont.com/2019/03/28/nyc-parking-garage-nyc-self-park-garages-staten-island-self-storage/
    http://8.remmont.com/2019/04/01/dallas-web-design-company-bottle-rocket-corp-web-design-company-dallas/
    http://6.remmont.com/2019/04/01/fredericksburg-christian-school-fredericksburg-christian-school-fredericksburg-colleges/
    http://5.remmont.com/2019/04/02/california-address-and-phone-number-video-contact-caltrans/
    http://7.remmont.com/2019/03/29/1-reverse-mortgage-calculator-age-reverse-mortgages-calculator/
    http://10.remmont.com/2019/04/01/florida-board-of-nursing-licensing-renewals-information-licensed-practical-nurse-programs/
    http://9.remmont.com/2019/03/28/water-damage-restoration-costs/
    http://2.remmont.com/2019/04/18/insurance-broker-barrie-video/
    http://9.remmont.com/2019/04/19/personal-trainer-insurance-australia-video/
    http://4.remmont.com/2019/04/01/nationwide-term-life-insurance-video/
    http://1.remmont.com/2019/04/ak-alaska-phone-number-video/
    http://9.remmont.com/2019/03/23/truck-rental-houston/
    http://10.remmont.com/2019/03/30/my-credit-score-video/
    http://5.remmont.com/2019/03/24/photographers-in-chicago/
    http://4.remmont.com/2019/03/30/how-can-i-check-my-credit-rating-for-free-video/
    http://8.remmont.com/2019/04/08/%e5%ae%89%e3%81%84%e5%8c%bb%e7%99%82%e4%bf%9d%e9%99%ba-video/
    http://9.remmont.com/2019/03/31/treemont-nursing-home-and-rehabilitation/
    http://10.remmont.com/2019/04/09/self-therapy-changing-your-personality-alcoholic-therapy/
    http://9.remmont.com/2019/03/24/what-number-is-florida-state-video/
    http://9.remmont.com/2019/03/30/direct-truck-insurance-provides-cargo-fleet-semi-truck-icc-authority-for-truckers-direct-insurance-quotes/
    http://9.remmont.com/2019/04/01/museums-in-dallas-museums-in-dallas/
    http://7.remmont.com/2019/04/01/insurance-policy-for-pregnant-women-video/
    http://5.remmont.com/2019/04/06/coretech-blog-blog-archive-system-center-2012-configuration-manager-sql-recommendations/
    http://7.remmont.com/2019/03/28/pre-settlement-funding-loans-lawsuit-cash-advances-lawsuit-loans/
    http://8.remmont.com/2019/04/05/why-is-florida-named-florida-video-why-is-florida-named-florida/
    http://1.remmont.com/2019/04/abogados-en-madrid-bufete-internacional-de-abogados-abogados-laboralistas-madrid/
    http://4.remmont.com/2019/04/05/agencias-de-seguros-video-ver-empresas-agencias-de-seguros-en-espa%d1%81a/
    http://9.remmont.com/2019/03/29/graduate-program-education/
    http://8.remmont.com/2019/04/09/seattle-auto-accident-car-accident-attorneys/
    http://3.remmont.com/2019/03/20/san-diego-beach-resorts-video/
    http://3.remmont.com/2019/04/07/wireless-home-alarms-at/
    http://2.remmont.com/2019/03/25/ngs-american-health-insurance-video/
    http://8.remmont.com/2019/04/02/cotizar-seguro-video-cotiza-tu-seguro-de-auto-en-linea-comparador-de-seguros-arca/
    http://7.remmont.com/2019/04/06/san-diego-family-law-attorney-la-jolla-divorce-lawyer/
    http://2.remmont.com/2019/04/17/insurance-quotes-for-small-businesses-video/
    http://9.remmont.com/2019/04/19/all-risks-insurance-brokers-video/
    http://7.remmont.com/2019/03/31/how-to-create-a-system-image-backup-in-windows-7-8-or-10/
    http://3.remmont.com/2019/04/05/personal-injury-attorneys-2/
    http://3.remmont.com/2019/04/05/flying-ants-winged-termites-winged-ants-flying-termite/
    http://9.remmont.com/2019/04/08/term-life-insurance-quotes-no-email-required-to-view-instant-rates-term-life-insurance-without-physical/
    http://10.remmont.com/2019/04/07/risikolebensversicherung-ohne-gesundheitsfragen-video/
    http://10.remmont.com/2019/04/02/medicare-weight-loss-surgery-all-you-need-to-know-bariatric-surgery-source/
    http://7.remmont.com/2019/03/29/la-habra-plumbers/
    http://4.remmont.com/2019/04/05/unc-wolfpack-and-video/
    http://10.remmont.com/2019/04/01/how-crew-died-according-to-nasa-study/
    http://6.remmont.com/2019/04/08/what-s-the-difference-between-vpn-and-cloud-computing/
    http://7.remmont.com/2019/04/08/what-is-siem-security-incident-and-event-manager-webopedia-definition/
    http://8.remmont.com/2019/04/08/how-to-open-a-roth-ira-account-11-steps-with-pictures-roth-ira-changes/
    http://8.remmont.com/2019/03/25/houston-real-estate-market-2017-flooding-video/
    http://3.remmont.com/2019/03/22/budget-used-cars-in-aberdeen-wa-360-637-4418-low-cost-used-cars-video/
    http://6.remmont.com/2019/04/05/diva-cloud-service/
    http://5.remmont.com/2019/04/12/northern-de-video/
    http://10.remmont.com/2019/03/30/georgia-state-university-website-video/
    http://4.remmont.com/2019/03/31/online-mph-degree-master-of-public-health-college-of-public-health-and-health-professions-university-of-florida-master-degree-paper/
    http://4.remmont.com/2019/03/31/swic-lpn-to-rn-program/
    http://8.remmont.com/2019/04/20/sole-trader-public-liability-insurance-video/
    http://10.remmont.com/2019/03/24/how-batman-v-superman-dawn-of-justice-should-have-ended-video/
    http://7.remmont.com/2019/04/03/nuclear-fusion-definition-of-nuclear-fusion-by-the-free-dictionary/
    http://6.remmont.com/2019/04/03/what-is-a-short-sale-definition-and-meaning/
    http://5.remmont.com/2019/04/03/car-insurance-quotes-washington-state-video-cheapest-car-insurance-in-washington/
    http://2.remmont.com/2019/04/19/cheap-beach-umbrella-video/
    http://5.remmont.com/2019/04/01/3-credit-reports-video-get-your-3-in-1-credit-report-and-free-credit-score-online/
    http://10.remmont.com/2019/04/04/multiple-system-atrophy-background-etiology-and-pathophysiology-epidemiology-define-erectile-dysfunction/
    http://9.remmont.com/2019/04/01/security-guard-services-in-los-angeles/
    http://4.remmont.com/2019/04/05/lowest-car-insurance-rates-in-ny-video-best-car-insurance-rates-in-buffalo-ny-quotewizard/
    http://4.remmont.com/2019/03/22/5-steps-to-becoming-a-millionaire-grant-cardone-trains-his-sales-team-live-video/
    http://5.remmont.com/2019/04/01/what-are-the-top-nursing-schools-in-saint-louis-mo/
    http://8.remmont.com/2019/04/20/cheap-landlord-house-insurance-video/
    http://7.remmont.com/2019/04/01/business-liability-insurance-quotes-online-video/
    http://3.remmont.com/2019/04/02/the-benefits-of-a-free-web-based-ehr/
    http://4.remmont.com/2019/04/10/inmobilien-inmobilien/
    http://2.remmont.com/2019/04/06/st-louis-personal-injury-lawyer/
    http://4.remmont.com/2019/04/21/insurance-broker-quotes-video/
    http://3.remmont.com/2019/04/08/how-to-buy-a-stock/
    http://8.remmont.com/2019/04/17/personal-workers-compensation-insurance-video/
    http://9.remmont.com/2019/04/03/credit-repair-houston-homepage-texas-credit-fix-texas-credit-repair-btcp-llc/
    http://5.remmont.com/2019/04/06/real-estate-australia-video/
    http://2.remmont.com/2019/04/17/6-months-car-insurance-policy-video/
    http://10.remmont.com/2019/03/30/tuck-school-of-business-dartmouth-college-mba-the-best-business-school-in-the-world/
    http://4.remmont.com/2019/04/07/bond-yield-to-maturity-calculator-for-comparing-bonds-bond-maturity/
    http://2.remmont.com/2019/03/21/insurance-assurance/
    http://2.remmont.com/2019/04/18/florida-automobile-insurance-company-video/
    http://7.remmont.com/2019/03/25/independent-life-insurance-agent-association-video/
    http://10.remmont.com/2019/04/08/gastric-bypass-sleeve-gastrectomy-surgery-cost/
    http://6.remmont.com/2019/04/01/government-nurse-practitioner-jobs/
    http://3.remmont.com/2019/03/21/sell-my-house-dont-make-this-mistake-mike-cindy-jones-realtors-904-874-0422-video/
    http://7.remmont.com/2019/03/31/california-family-health-insurance-quote-video/
    http://2.remmont.com/2019/04/05/university-of-missouri-usa-video/
    http://2.remmont.com/2019/04/18/become-independent-insurance-broker-video/
    http://4.remmont.com/2019/04/09/principales-companias-que-ofrecen-seguro-medico-en-estados-unidos-seguro-m%d0%b3dico-online-empresas-de-seguros/
    http://9.remmont.com/2019/04/08/hochzeitsdeko-weddstyle-verleih-von-stuhlhussen/
    http://7.remmont.com/2019/04/02/sf-explores-blazing-fast-gigabit-internet-connections-san-francisco-chronicle/
    http://8.remmont.com/2019/04/03/free-medicare-supplemental-insurance-video-get-medicare-supplemental-insurance-plan-quotes-find-medigap-coverage/
    http://5.remmont.com/2019/03/21/movers-san-diego-video/
    http://10.remmont.com/2019/04/01/efs-data-recovery-for-windows-7/
    http://7.remmont.com/2019/03/30/how-to-get-an-online-bachelor-degree-in-nursing/
    http://4.remmont.com/2019/04/20/general-liability-insurance-nj-video/
    http://8.remmont.com/2019/04/01/2003-ford-mustang-svt-cobra-price-review-road-test-motor-trend/
    http://4.remmont.com/2019/03/25/united-india-motor-insurance-online-renewal-video/
    http://1.remmont.com/2019/03/colleges-in-marietta/
    http://2.remmont.com/2019/04/18/horse-liability-insurance-video-2/
    http://6.remmont.com/2019/04/06/alcohol-and-drug-services-ads/
    http://8.remmont.com/2019/04/16/best-personal-dental-insurance-video/
    http://5.remmont.com/2019/04/01/pet-insurance-comparison-ireland-video-compare-pet-insurance-northern-ireland/
    http://4.remmont.com/2019/04/21/long-term-health-care-insurance-rates-video/
    http://5.remmont.com/2019/04/05/nebraska-personal-property-tax-return-video/
    http://2.remmont.com/2019/04/08/colorado-vehicle-insurance-requirements-co-dmv-vehicle-insurance-information-colorado-automobile-insurance/
    http://10.remmont.com/2019/04/03/network-flow-programming-models/
    http://4.remmont.com/2019/03/23/los-angeles-tour-packages/
    http://7.remmont.com/2019/03/29/audi-repair-automotive-services/
    http://8.remmont.com/2019/03/19/hello-world/
    http://2.remmont.com/2019/04/17/chesapeake-personal-property-tax-video/
    http://4.remmont.com/2019/04/05/best-auto-attendant-systems-2017-reviews-pricing-demos-voip-auto-attendant/
    http://7.remmont.com/2019/04/02/denver-car-insurance-agents/
    http://10.remmont.com/2019/04/06/alcohol-rehab-treatment-centers-orlando-fl-407-358-6225/
    http://2.remmont.com/2019/04/19/citizens-insurance-group-video/
    http://3.remmont.com/2019/03/23/filing-your-taxes-online-how-to-with-turbo-tax-video/
    http://10.remmont.com/2019/03/22/car-rental-christchurch-airport-video/
    http://9.remmont.com/2019/04/02/secretary-of-state-downtown-chicago-video-chicago-illinois-apostille/
    http://7.remmont.com/2019/04/02/tennessee-auto-insurance-laws-regulations/
    http://10.remmont.com/2019/03/28/conduct-online-surveys-using-opinio-customer-service-tools/
    http://8.remmont.com/2019/04/21/australian-flood-maps-video/
    http://7.remmont.com/2019/04/07/accused-of-a-washington-state-drug-or-vucsa-crime/
    http://8.remmont.com/2019/04/03/any-address-in-california-video-privacy-laws-state-of-california-department-of-justice-office-of-the-attorney-general/
    http://2.remmont.com/2019/04/02/roland-versicherung-video-roland-rechtsschutz-angebot-anfordern-gothaer/
    http://2.remmont.com/2019/04/06/mortgage-interest-rates-video/
    http://4.remmont.com/2019/04/04/psyd-in-psychology-online-programs/
    http://3.remmont.com/2019/04/04/school-of-literature-drama-and-creative-writing-graduate-school-creative-writing/
    http://1.remmont.com/2019/03/s-r-rooter-service-is-a-plumber-in-perris-ca/
    http://8.remmont.com/2019/04/05/robinhood-insurance-broker-pvt-ltd-video-robinhood-insurance-broker-pvt-ltd/
    http://4.remmont.com/2019/04/03/bachelors-degree-in-early-childhood-education/
    http://10.remmont.com/2019/04/02/medical-insurance-for-visitors-to-canada/
    http://7.remmont.com/2019/04/06/essential-nursing-and-care-agency/
    http://4.remmont.com/2019/03/31/viking-refrigerator-repair-service/
    http://3.remmont.com/2019/03/30/versicherungen-fur-selbststandige-video/
    http://9.remmont.com/2019/03/27/small-group-1-video/
    http://9.remmont.com/2019/04/17/cheap-irish-car-insurance-video/
    http://10.remmont.com/2019/04/01/bankruptcy-information-learn-more/
    http://3.remmont.com/2019/03/28/missouri-health-insurance-individual-market-2018-participants-video/
    http://5.remmont.com/2019/03/24/los-angeles-college/
    http://6.remmont.com/2019/03/24/university-of-il-champaign-video/
    http://4.remmont.com/2019/04/05/non-nursing-bachelor-s/
    http://3.remmont.com/2019/04/07/kerio-secure-business-collaboration-home-kerio-technologies-corporate-firewall/
    http://7.remmont.com/2019/03/24/los-angeles-apartments/
    http://10.remmont.com/2019/04/01/peninsula-orthopaedic-associates/
    http://1.remmont.com/2019/04/best-travel-websites-video/
    http://2.remmont.com/2019/04/19/compare-house-insurance-prices-video-2/
    http://6.remmont.com/2019/03/26/homeowners-insurance-how-is-the-dwelling-coverage-figured-video/
    http://7.remmont.com/2019/04/02/community-and-human-services/
    http://6.remmont.com/2019/03/23/questions-answered-about-fannie-mae-homes-for-sale-and-homepath-video/
    http://7.remmont.com/2019/04/02/international-news-latest-world-news-videos-photos-abc-news-abc-news-type-of-government-in-russia/
    http://7.remmont.com/2019/03/29/research-for-neurobiochemistry-and-neurological-disease-wako-pure-chemical-industries-ltd/
    http://2.remmont.com/2019/03/24/types-of-general-insurance-video/
    http://8.remmont.com/2019/04/02/cloud-automation-managed-services/
    http://4.remmont.com/2019/03/21/best-bad-credit-lenders-unsecured-personal-loans-in-2019-video/
    http://4.remmont.com/2019/04/19/should-i-get-private-health-insurance-video/
    http://4.remmont.com/2019/04/04/nada-personal-watercraft-video/
    http://9.remmont.com/2019/03/25/home-and-contents-insurance-flood-cover-and-video/
    http://2.remmont.com/2019/03/21/insurance-101-personal-umbrella-policy-video/
    http://3.remmont.com/2019/04/06/applicant-tracking-system/
    http://6.remmont.com/2019/03/30/versicherung-24-video/
    http://10.remmont.com/2019/04/03/how-i-change-the-thickness-of-my-hr-tag-stack-overflow/
    http://6.remmont.com/2019/03/23/things-to-do-in-phoenix/
    http://7.remmont.com/2019/03/31/west-palm-beach-car-accident-lawyers-local-attorneys-law-firms-in-west-palm-beach-fl/
    http://5.remmont.com/2019/04/03/zuerich-versicherung-and-video-zurich-versicherungs-aktiengesellschaft-as-an-employer-xing-companies/
    http://1.remmont.com/2019/03/assurance-vehicule-video/
    http://9.remmont.com/2019/04/13/wheres-virginia-and-video/
    http://6.remmont.com/2019/03/21/how-to-replace-front-strut-03-11-honda-element-video/
    http://5.remmont.com/2019/04/07/new-york-plastic-surgery/
    http://5.remmont.com/2019/04/06/nti-career-school-medical-it-hvac-cdl-programs/
    http://8.remmont.com/2019/04/07/2014-mba-rankings-best-business-schools/
    http://1.remmont.com/2019/03/retrolegends-classic-and-sportscars/
    http://6.remmont.com/2019/04/08/viajes-y-viajes-agencias-de-viajes/
    http://7.remmont.com/2019/04/02/7-cloud-based-database-services/
    http://7.remmont.com/2019/04/05/orlando-dui-lawyer-reveals/
    http://10.remmont.com/2019/04/01/how-to-form-an-llc-in-delaware/
    http://7.remmont.com/2019/03/24/airports-in-los-angeles-area/
    http://6.remmont.com/2019/03/30/attbusiness-net/
    http://3.remmont.com/2019/04/09/master-s-in-psychology-degree-program-online-capella-university/
    http://1.remmont.com/2019/04/remote-medical-billing-jobs-video/
    http://4.remmont.com/2019/04/17/missouri-disability-application-video/
    http://6.remmont.com/2019/03/24/multi-trip-travel-medical-insurance-options-video/
    http://10.remmont.com/2019/04/04/detox-facility-rapid-drug-detox/
    http://1.remmont.com/2019/03/los-angeles-hostels/
    http://4.remmont.com/2019/03/30/seguros-pelayo-video/
    http://2.remmont.com/2019/04/11/einhorn-apotheke-58452-witten-apotheke-erle/
    http://1.remmont.com/2019/04/airlines-video/
    http://2.remmont.com/2019/03/30/kraftfahrzeug-versicherung-video/
    http://9.remmont.com/2019/03/23/of-illinois-video/
    http://5.remmont.com/2019/04/09/bem-vindo-ao-patrocinio-magico-patrocinio-magico-multinivel/
    http://10.remmont.com/2019/03/28/denver-flat-roofing-commercial-roofing-denver-co-roofing-co/
    http://1.remmont.com/2019/04/age-concern-home-insurance-video/
    http://6.remmont.com/2019/04/04/low-cost-auto-insurance-in-california/
    http://6.remmont.com/2019/04/06/optimum-vision-insurance-video/
    http://4.remmont.com/2019/04/10/franchise-cession-reprise-franchise-ou-cession-de-franchise-entreprise-en-franchise-%d0%b0-reprendre-franchise/
    http://6.remmont.com/2019/04/05/georgias-video-georgia-s-sea-creatures-georgia-public-broadcasting/
    http://8.remmont.com/2019/04/20/lowest-term-life-insurance-rates-video/
    http://8.remmont.com/2019/04/01/solar-sales-proposal-software-solar-electric-pv-energy-storage-battery-solar-water-thermal-energy-efficiency-wind-internet-marketing-proposal/
    http://10.remmont.com/2019/04/07/assurance-temporaire-voiture-and-video/
    http://10.remmont.com/2019/04/04/personal-injury-attorney-in-ocala-fl/
    http://3.remmont.com/2019/04/09/cheyenne-mountain-dental-group/
    http://8.remmont.com/2019/04/01/go-to-arizona-video-roosevelt-lake/
    http://2.remmont.com/2019/04/17/national-insurance-company-claim-form-video/
    http://8.remmont.com/2019/03/22/mla-in-text-citations-step-by-step-guide-video/
    http://5.remmont.com/2019/04/07/voip-gateways-from-patton/
    http://9.remmont.com/2019/03/31/fast-university-degrees-in-days/
    http://9.remmont.com/2019/03/24/texas-state-records-video/
    http://3.remmont.com/2019/04/08/50-ideas-for-things-to-do-in-louisiana-this-summer/
    http://1.remmont.com/2019/04/long-term-care-facilities-in-ottawa-video/
    http://9.remmont.com/2019/04/06/small-personal-loans-online-video/
    http://8.remmont.com/2019/04/02/best-interest-free-deals-our-favourite-balance-transfer-credit-cards/
    http://5.remmont.com/2019/03/25/affordable-taxi-insurance-video/

  3. [url=http://remmont.com/category/rental/]property development finance
    [/url]
    Foreclosures in $ Video
    Foreclosures in Foreclosures in Maryland Foreclosure Listings Maryland foreclosure listings of foreclosures for sale and nationwide. Our foreclosure data includes Maryland bank owned homes, foreclosed homes and REO properties. All our foreclosure listings are screened for accuracy by our team of professionals on a daily basis. All of our Maryland foreclosure homes for sale will also include the contact information, number of beds and baths, SQFT and price. Top 10 Cities Maryland Latest Resources Baltimore, referred to as the Charm City, is United States’ . Maryland Foreclosures for Sale The state of Maryland allows both judicial and non-judicial foreclosures against …
    The post [url=http://finances.remmont.com/foreclosures-in-video/]Foreclosures in $ Video[/url] appeared first on [url=http://finances.remmont.com] Finances[/url] .
    [url=http://remmont.com/]vinson mortgage
    [/url]
    [url=http://asca5.tripod.com/Gaestebog.htm] full credit check [/url]
    [url=http://ozakit.o.oo7.jp/cgi-bin/ezmixed.cgi] fico score report free [/url]
    [url=http://52ptw.com/home.php?mod=space&uid=40450] free report [/url]
    [url=http://django-knowledge.org/questions/2/this-is-a-public-question/] bank of america starter credit card [/url]
    [url=https://www.dver-nik.ru/products/dveri-pdg-2109-ubertyure#comment_5293/] check on credit one application [/url]

    [u][b]BONUS 250 FREE LINKS SITE & BLOG [/b][/u]
    [url=http://realestate.remmont.com/blue-ridge-north-georgia-real-estate-georgia-real-estate-georgia-real-estate-2/]Blue Ridge North Georgia Real Estate, georgia real estate.[/url]
    [url=http://realestate.remmont.com/washing-machine-repair-singapore-services-washing-machine-repair-in-singapore-washing-machine-repair-service-in-singaporecheap-washing-machine-repair-singapore-appliance-repair-wa/]Washing Machine Repair Singapore Services[/url]
    [url=http://realestate.remmont.com/the-official-mls-in-cozumel-quintana-roo-mexico-ocala-real-estate/]The Official MLS in Cozumel, Quintana Roo, MEXICO[/url]
    [url=http://realestate.remmont.com/historic-brooklyn-heights-townhouse-brooklyn-ny-townhouse-new-york-city-real-estate-real-estate-russia/]Historic Brooklyn Heights Townhouse, Brooklyn NY Townhouse – New York City Real Estate[/url]
    [url=http://realestate.remmont.com/visit-st-simons-island-georgia-the-official-travel-tourism-guide-to-st-simons-island-ga-find-hotels-vacation-rentals-events-shopping-restaurants-real-estate-real-estate-signs/]Visit St. Simons Island Georgia! The Official Travel – Tourism Guide to St. Simons Island, GA: Find Hotels, Vacation Rentals, Events, Shopping, Restaurants [/url]

Leave a Reply

Your email address will not be published. Required fields are marked *