2015-07-21 19:40:16.089 18733 ERROR nova.api.openstack [req-54fc4923-c02b-45f4-bc6e-9389618e2ce3 None] Caught error: Timed out waiting for a reply to message ID 0dc42ab20d774743b02e7d4c0c2bc267
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack Traceback (most recent call last):
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 124, in __call__
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     return req.get_response(self.application)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     application, catch_exc_info=False)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     return resp(environ, start_response)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 663, in __call__
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     return self.app(env, start_response)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     return resp(environ, start_response)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     return resp(environ, start_response)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     response = self.app(environ, start_response)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     return resp(environ, start_response)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 910, in __call__
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     content_type, body, accept)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 975, in _process_stack
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1059, in dispatch
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     return method(req=request, **action_args)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 959, in create
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     check_server_group_quota=check_server_group_quota)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 131, in inner
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     rv = f(*args, **kwargs)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1457, in create
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     check_server_group_quota=check_server_group_quota)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1079, in _create_instance
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     max_count)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 770, in _validate_and_build_base_options
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     requested_networks, max_count)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 479, in _check_requested_networks
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     max_count)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 48, in wrapped
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     return func(self, context, *args, **kwargs)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 404, in validate_networks
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     requested_networks)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 245, in validate_networks
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     return self.client.call(ctxt, 'validate_networks', networks=networks)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 389, in call
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     return self.prepare().call(ctxt, method, **kwargs)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     retry=self.retry)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     timeout=timeout, retry=retry)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     retry=retry)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 397, in _send
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     result = self._waiter.wait(msg_id, timeout)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 285, in wait
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     reply, ending = self._poll_connection(msg_id, timeout)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 235, in _poll_connection
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack     % msg_id)
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack MessagingTimeout: Timed out waiting for a reply to message ID 0dc42ab20d774743b02e7d4c0c2bc267
2015-07-21 19:40:16.089 18733 TRACE nova.api.openstack
2015-07-21 19:40:16.102 18733 INFO nova.api.openstack [req-54fc4923-c02b-45f4-bc6e-9389618e2ce3 None] http://phl-controller-01.fabric:8774/v2/4680107a78db4ac58a5880f95e95ee30/servers returned with HTTP 500
2015-07-21 19:40:16.106 18733 INFO nova.osapi_compute.wsgi.server [req-54fc4923-c02b-45f4-bc6e-9389618e2ce3 None] 10.1.1.162 "POST /v2/4680107a78db4ac58a5880f95e95ee30/servers HTTP/1.1" status: 500 len: 271 time: 62.4156260