AgentManager-SSLHandshakeHandler SSL error

[sotheareth]

1. I try to build and run apache cloudstack in my local
   from https://github.com/apache/cloudstack and
   use branch either 4.17-user-shared-networks or main I got below error in step 3

2. follow these step until the run jetty step under Build CloudStack
   
   $ mvn -pl :cloud-client-ui jetty:run # Run the management server on port 8080
   
   from https://cwiki.apache.org/confluence/display/CLOUDSTACK/Setting+up+CloudStack+Development+Environment+on+Linux

3. getting below error:
   
   ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-5:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.0.104:8250, remote address=/192.168.0.104:56606.
   WARN [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-11:null) (logid:) This SSL engine was forced to close inbound due to end of stream.
   javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify
   at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
   at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
   at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:336)
   at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
   at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:283)
   at java.base/sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:733)
   at com.cloud.utils.nio.Link.doHandshakeUnwrap(Link.java:490)
   at com.cloud.utils.nio.Link.doHandshake(Link.java:618)
   at com.cloud.utils.nio.NioConnection$1.run(NioConnection.java:216)
   at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   at java.base/java.lang.Thread.run(Thread.java:829)
   ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-11:null) (logid:) Failed to send server's CLOSE message due to socket channel's failure.
   INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-7:null) (logid:) Connection from /192.168.0.104 closed but no cleanup was done.
   
   
   ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-11:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.0.104:8250, remote address=/192.168.0.104:56610.
   WARN [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-4:null) (logid:) This SSL engine was forced to close inbound due to end of stream.
   javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify
   at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
   at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
   at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:336)
   at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
   at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:283)
   at java.base/sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:733)
   at com.cloud.utils.nio.Link.doHandshakeUnwrap(Link.java:490)
   at com.cloud.utils.nio.Link.doHandshake(Link.java:618)
   at com.cloud.utils.nio.NioConnection$1.run(NioConnection.java:216)
   at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   at java.base/java.lang.Thread.run(Thread.java:829)
   ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-4:null) (logid:) Failed to send server's CLOSE message due to socket channel's failure.
   INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-8:null) (logid:) Connection from /192.168.0.104 closed but no cleanup was done.
   ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-5:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.0.104:8250, remote address=/192.168.0.104:56612.
   ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-5:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.0.104:8250, remote address=/192.168.0.104:56616.
   ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-4:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.0.104:8250, remote address=/192.168.0.104:56618.
   WARN [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-9:null) (logid:) This SSL engine was forced to close inbound due to end of stream.
   javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify
   at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
   at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
   at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:336)
   at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
   at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:283)
   at java.base/sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:733)
   at com.cloud.utils.nio.Link.doHandshakeUnwrap(Link.java:490)
   at com.cloud.utils.nio.Link.doHandshake(Link.java:618)
   at com.cloud.utils.nio.NioConnection$1.run(NioConnection.java:216)
   at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   at java.base/java.lang.Thread.run(Thread.java:829)
   ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-9:null) (logid:) Failed to send server's CLOSE message due to socket channel's failure.

I stucked and I'm new to apache cloudstack and my environments: ubuntu 20.04.

Can anyone suggest me how to fix that issue?

[yadvr]

@sotheareth are you running KVM agent (cloudstack-agent) on the same host? Can you redeploy your dev-test env, or re-add the host with ssh allowed on root user using password (in sshd_config you've to permit root login by password). Alternatively, you can also explore using mbx: https://github.com/shapeblue/mbx
The specific user-shared network branch may not be ready for testing yet, cc @weizhouapache

[sotheareth]

@rohityadavcloud

• I installed cloudstack-agent on the same host and
• already allow sshd_config for root:
  PermitRootLogin yes and PasswordAuthentication yes
• I did not try your https://github.com/shapeblue/mbx yet

[yadvr]

@sotheareth Assuming you're doing something for testing, you can set the ca.plugin.root.auth.strictness global setting to false temporarily (this doesn't require restarting mgmt server) and then go to UI->Infra->Hosts-> select the host -> click the provision certificate button (this will re-initiatlise the host certificates). Then you can change the auth strictness global setting to true again and restart cloudstack-agent.
The possible cause for this issue is usually that when adding the KVM host the certificates for some reason weren't correctly setup (you may check logs to investigate further).

[sotheareth]

@rohityadavcloud

• I'm not sure if run from source code it requires to install cloudstack-agent or is there any cloudstack-agent startup when I run from jetty.
• after I change ca.plugin.root.auth.strictness = false and it works fine
• I can add secondary storage - success
• now I have another issue
  I cannot add new host - error said 'Error 530 Unable to add the host'
  and I check there is no any record for host table in datatabase

in the log file:
021-12-29 23:33:26,372 WARN [o.a.c.a.c.a.h.AddHostCmd] (qtp1742961957-284:ctx-7fc26eee ctx-a2015a30) (logid:228e7018) Exception:
com.cloud.exception.DiscoveryException: Unable to add the host
at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:878)
at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:641)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy210.discoverHosts(Unknown Source)
at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:142)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:772)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:596)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131)
at com.cloud.api.ApiServlet.doPost(ApiServlet.java:98)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1386)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:226)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:829)

* Zone Name : primary-zone
* Pod name: primary-pod
* Cluster name: primary-cluster
* Host: 192.168.100.105
* Username: root
* Password: ...

• how does cloudstack look up the host?

[nvazquez]

Hi @sotheareth - the cloudstack-agent package needs to be installed on your KVM host (assuming you are using KVM). Also, check if the port 8250 is open as it is the port the management server and the host agent connect to

[yadvr]

@sotheareth were you able to get help? If not can you join our dev mailing list and continue there -
https://cloudstack.apache.org/mailing-lists.html

Or additional resources:
https://github.com/shapeblue/hackerbook/blob/main/2-dev.md