Block storage addition issues on XCP-ng

[DagSonsteboSB]

ISSUE TYPE

• Bug Report

COMPONENT NAME

• createStoragepool API
• iSCSI storage
• issue may also affect FC - TBC
• XCP-ng

CLOUDSTACK VERSION

• Tested with ACS 4.11.2

CONFIGURATION

• Advanced zone

OS / ENVIRONMENT

• Mgmt on CentOS7
• XCP-ng version 7.4

SUMMARY

• createStoragepool with the iSCSI / defaultStoraprovider will successfully add iSCSI pools to the XCP-ng hosts but not configure the pool furher in CloudStack
• createStoragepool with Presetup / defaultStorageprovider will behave similarly - i.e. not handle the actual addition of the primary pools to CloudStack

STEPS TO REPRODUCE

• Create advanced zone with XCP-ng 7.x hosts
• add iSCSI primary storage
• observe createStoragepool behaviour

EXPECTED RESULTS

• iSCSI pool successfully added

ACTUAL RESULTS

• iSCSI pools added successfully as SRs to the XCP-ng hosts but not to CloudStack

Log excerpt

2019-04-18 08:45:01,287 WARN  [c.c.h.x.r.w.x.CitrixModifyStoragePoolCommandWrapper] (DirectAgent-5:ctx-0f96ece5) (logid:70745797) ModifyStoragePoolCommand add XenAPIException:Unable to setup heartbeat sr on SR 55fddcb5-b095-e685-ba47-9388e120d5db due to #7# volume hb-a615cbc3-cb8d-47f4-b9c6-271a29e6defd is not created host:a615cbc3-cb8d-47f4-b9c6-271a29e6defd pool: 10.10.10.10/iqn.2017-01.local.sblab:1faba1ed-79fc-46a3-acc6-4a60a7dcd65f/0
com.cloud.utils.exception.CloudRuntimeException: Unable to setup heartbeat sr on SR 55fddcb5-b095-e685-ba47-9388e120d5db due to #7# volume hb-a615cbc3-cb8d-47f4-b9c6-271a29e6defd is not created
        at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.setupHeartbeatSr(CitrixResourceBase.java:4743)
        at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixModifyStoragePoolCommandWrapper.execute(CitrixModifyStoragePoolCommandWrapper.java:54)
        at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixModifyStoragePoolCommandWrapper.execute(CitrixModifyStoragePoolCommandWrapper.java:40)
        at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
        at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1737)
        at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2019-04-18 08:45:01,288 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-5:ctx-0f96ece5) (logid:70745797) Seq 1-4089831411605831693: Response Received:
2019-04-18 08:45:01,288 DEBUG [c.c.a.t.Request] (DirectAgent-5:ctx-0f96ece5) (logid:70745797) Seq 1-4089831411605831693: Processing:  { Ans: , MgmtId: 32986842072929, via: 1(ref-trl-1050-x-M7-dsonstebo-xs1), Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":false,"details":"ModifyStoragePoolCommand add XenAPIException:Unable to setup heartbeat sr on SR 55fddcb5-b095-e685-ba47-9388e120d5db due to #7# volume hb-a615cbc3-cb8d-47f4-b9c6-271a29e6defd is not created host:a615cbc3-cb8d-47f4-b9c6-271a29e6defd pool: 10.10.10.10/iqn.2017-01.local.sblab:1faba1ed-79fc-46a3-acc6-4a60a7dcd65f/0","wait":0}}] }

[conradrock]

Just to note, I observed this exact same behavior. I was searching to determine a fix, but I'm on XCP-ng 7.6, adding iSCSI block storage, works fine with the XCP-ng 7.6 hosts directly, but I can't add block storage to CloudStack. Here's an example error, numerous others follow.

2019-06-09 22:59:53,885 WARN [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp2114650936-334:ctx-09e3deab ctx-ec4eecc3) (logid:b61131ba) Unable to establish a connection between Host[-2-Routing] and org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@13 com.cloud.utils.exception.CloudRuntimeException: Unable establish connection from storage head to storage pool 3 due to ModifyStoragePoolCommand add XenAPIException:Unable to setup heartbeat sr on SR 0133904e-aef7-e0ae-0f53-ca3641bd9034 due to #7# volume hb-bc60985d-8c02- at org.apache.cloudstack.storage.datastore.provider.DefaultHostListener.hostConnect(DefaultHostListener.java:76) at com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:951) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy94.connectHostToSharedPool(Unknown Source) at org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.attachCluster(CloudStackPrimaryDataStoreLifeCycleImpl.java:414) at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:722) at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:202) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy94.createPool(Unknown Source) at org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:163) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156) at com.cloud.api.ApiServer.queueCommand(ApiServer.java:758) at com.cloud.api.ApiServer.handleRequest(ApiServer.java:582) at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:310) at com.cloud.api.ApiServlet$1.run(ApiServlet.java:130) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:127) at com.cloud.api.ApiServlet.doGet(ApiServlet.java:89) at javax.servlet.http.HttpServlet.service(HttpServlet.java:686) at javax.servlet.http.HttpServlet.service(HttpServlet.java:791) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:852) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:527) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:530) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748)

[msinhore]

The same issue happen with HBA LUNs.