Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How can we troubleshoot OutOfMemoryError in Pinpoint 3.0 Collector Component? #11983

Open
1inuxboy opened this issue Jan 20, 2025 · 2 comments
Open

How can we troubleshoot OutOfMemoryError in Pinpoint 3.0 Collector Component? #11983

1inuxboy opened this issue Jan 20, 2025 · 2 comments

Comments

@1inuxboy
Copy link

What version of pinpoint are you using?

Pinpoint 3.0

Describe your problem

I have deployed Pinpoint 3.0 using the official pinpoint-docker repository with docker-compose. The deployment seems successful, as all containers are running without errors. However, no trace data is visible in the Web UI after attempting to trace applications with the agent.

What have you done?

This issue does not occur 100% of the time. After I restart the collector service, it can recover briefly, but then the problem will reappear.

Configuration Details

  • Host OS: CentOS 7.9
  • Host Specs: 8-core CPU, 32 GB RAM
  • Docker Deployment: Official pinpoint-docker repository
  • Docker Compose File: Default configuration from the repository (no manual adjustments made).

Screenshots

Image

Logs

This is pinpoint-collector logs

01-20 07:28:15.015 [el-Worker(16-2)] INFO  i.g.n.N.connections                      : Transport failed
java.lang.OutOfMemoryError: Cannot reserve 32768 bytes of direct buffer memory (allocated: 10485032, limit: 10485760)
        at java.nio.Bits.reserveMemory(Unknown Source) ~[?:?]
        at java.nio.DirectByteBuffer.<init>(Unknown Source) ~[?:?]
        at java.nio.ByteBuffer.allocateDirect(Unknown Source) ~[?:?]
        at io.netty.buffer.UnpooledDirectByteBuf.allocateDirect(UnpooledDirectByteBuf.java:104) ~[netty-buffer-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.buffer.UnpooledDirectByteBuf.<init>(UnpooledDirectByteBuf.java:64) ~[netty-buffer-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:41) ~[netty-buffer-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:634) ~[netty-buffer-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:398) ~[netty-buffer-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) ~[netty-buffer-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[netty-buffer-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53) ~[netty-transport-native-unix-common-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:120) ~[netty-transport-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75) ~[netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:785) [netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509) [netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407) [netty-transport-classes-epoll-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.100.Final.jar:4.1.100.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.100.Final.jar:4.1.100.Final]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
01-20 07:28:15.015 [verExecutor4377] WARN  grpcAgentServerCallExecutor              : The executor uses finite bounds for both maximum threads and work queue capacity, and is saturated. Check the maxPoolSize, queueCapacity, and HBase options in the configuration. maxPoolSize=8, rejectedCount=6010
01-20 07:28:15.015 [ecycleFlusher-1] WARN  c.n.p.c.r.g.s.KeepAliveService           : Failed to update state. closeState:false lifeCycle=PingSession{id=4290, header=Header{name='agent', agentId='172.20.6.89', agentName='null', applicationName='***-record', agentStartTime=1737037610457, socketId=589, serviceType=1210, supportCommandCodeList=null, properties={}}, eventIdAllocator=2, serviceType=-1, updated=false, lastPingTimeMillis=1737355851800} Running/Agent ping
com.navercorp.pinpoint.common.hbase.HbaseSystemException: callTimeout=10000, callDuration=10433: Failed to get user region lock in10000 ms.  for accessing meta region server. row '172.20.6.89\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFEk}c\xBF\xA4' on table 'AgentEvent' at null
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate.rethrowHbaseException(HbaseTemplate.java:886) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate.execute(HbaseTemplate.java:854) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate.put(HbaseTemplate.java:240) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
        at com.navercorp.pinpoint.collector.dao.hbase.HbaseAgentEventDao.insert(HbaseAgentEventDao.java:85) ~[classes/:3.0.0]
        at jdk.internal.reflect.GeneratedMethodAccessor65.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:351) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) ~[spring-tx-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:717) ~[spring-aop-6.1.5.jar:6.1.5]
        at com.navercorp.pinpoint.collector.dao.hbase.HbaseAgentEventDao$$SpringCGLIB$$0.insert(<generated>) ~[classes/:3.0.0]
        at com.navercorp.pinpoint.collector.service.AgentEventService.insert(AgentEventService.java:44) ~[classes/:3.0.0]
        at jdk.internal.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:351) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:174) ~[spring-context-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:717) ~[spring-aop-6.1.5.jar:6.1.5]
        at com.navercorp.pinpoint.collector.service.AgentEventService$$SpringCGLIB$$0.insert(<generated>) ~[classes/:3.0.0]
        at com.navercorp.pinpoint.collector.service.async.AgentEventAsyncTaskService.handleEvent(AgentEventAsyncTaskService.java:51) ~[classes/:3.0.0]
        at com.navercorp.pinpoint.collector.receiver.grpc.service.KeepAliveService.updateState(KeepAliveService.java:98) ~[classes/:3.0.0]
        at com.navercorp.pinpoint.collector.receiver.grpc.service.KeepAliveService.updateState(KeepAliveService.java:59) ~[classes/:3.0.0]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
        at org.springframework.scheduling.support.ScheduledMethodRunnable.runInternal(ScheduledMethodRunnable.java:130) ~[spring-context-6.1.5.jar:6.1.5]
        at org.springframework.scheduling.support.ScheduledMethodRunnable.lambda$run$2(ScheduledMethodRunnable.java:124) ~[spring-context-6.1.5.jar:6.1.5]
        at io.micrometer.observation.Observation.observe(Observation.java:499) [micrometer-observation-1.12.4.jar:1.12.4]
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:124) [spring-context-6.1.5.jar:6.1.5]
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-6.1.5.jar:6.1.5]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
        at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.net.SocketTimeoutException: callTimeout=10000, callDuration=10433: Failed to get user region lock in10000 ms.  for accessing meta region server. row '172.20.6.89\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFEk}c\xBF\xA4' on table 'AgentEvent' at null
        at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:157) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.HTable.lambda$put$3(HTable.java:566) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:554) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate$5.doInTable(HbaseTemplate.java:243) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate$5.doInTable(HbaseTemplate.java:240) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate.execute(HbaseTemplate.java:852) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
        ... 45 more
Caused by: org.apache.hadoop.hbase.client.LockTimeoutException: Failed to get user region lock in10000 ms.  for accessing meta region server.
        at org.apache.hadoop.hbase.client.ConnectionImplementation.takeUserRegionLock(ConnectionImplementation.java:1133) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:996) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:897) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.HRegionLocator.lambda$getRegionLocation$0(HRegionLocator.java:80) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.HRegionLocator.tracedLocationFuture(HRegionLocator.java:162) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:79) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.RegionLocator.getRegionLocation(RegionLocator.java:71) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.RegionLocator.getRegionLocation(RegionLocator.java:60) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:219) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:101) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.HTable.lambda$put$3(HTable.java:566) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:554) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate$5.doInTable(HbaseTemplate.java:243) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate$5.doInTable(HbaseTemplate.java:240) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate.execute(HbaseTemplate.java:852) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
        ... 45 more
01-20 07:28:15.015 [CallExecutor445] WARN  c.n.p.c.r.g.s.KeepAliveService           : Failed to update state. closeState:false lifeCycle=PingSession{id=12817, header=Header{name='agent', agentId='***-ami-03', agentName='null', applicationName='***-ami', agentStartTime=1736870946684, socketId=1379, serviceType=1210, supportCommandCodeList=null, properties={}}, eventIdAllocator=1, serviceType=-1, updated=false, lastPingTimeMillis=1737358035714} Running/Agent connected
com.navercorp.pinpoint.common.hbase.HbaseSystemException: callTimeout=10000, callDuration=10433: Failed to get user region lock in10000 ms.  for accessing meta region server. row '***-ami-03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFEk}cx\xB5' on table 'AgentEvent' at null
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate.rethrowHbaseException(HbaseTemplate.java:886) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate.execute(HbaseTemplate.java:854) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate.put(HbaseTemplate.java:240) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
        at com.navercorp.pinpoint.collector.dao.hbase.HbaseAgentEventDao.insert(HbaseAgentEventDao.java:85) ~[classes/:3.0.0]
        at jdk.internal.reflect.GeneratedMethodAccessor65.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:351) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) ~[spring-tx-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:717) ~[spring-aop-6.1.5.jar:6.1.5]
        at com.navercorp.pinpoint.collector.dao.hbase.HbaseAgentEventDao$$SpringCGLIB$$0.insert(<generated>) ~[classes/:3.0.0]
        at com.navercorp.pinpoint.collector.service.AgentEventService.insert(AgentEventService.java:44) ~[classes/:3.0.0]
        at jdk.internal.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
        at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:351) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.validation.beanvalidation.MethodValidationInterceptor.invoke(MethodValidationInterceptor.java:174) ~[spring-context-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:765) ~[spring-aop-6.1.5.jar:6.1.5]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:717) ~[spring-aop-6.1.5.jar:6.1.5]
        at com.navercorp.pinpoint.collector.service.AgentEventService$$SpringCGLIB$$0.insert(<generated>) ~[classes/:3.0.0]
        at com.navercorp.pinpoint.collector.service.async.AgentEventAsyncTaskService.handleEvent(AgentEventAsyncTaskService.java:51) ~[classes/:3.0.0]
        at com.navercorp.pinpoint.collector.receiver.grpc.service.KeepAliveService.updateState(KeepAliveService.java:98) [classes/:3.0.0]
        at com.navercorp.pinpoint.collector.receiver.grpc.service.KeepAliveService.updateState(KeepAliveService.java:74) [classes/:3.0.0]
        at com.navercorp.pinpoint.collector.receiver.grpc.service.AgentLifecycleListener.connect(AgentLifecycleListener.java:61) [classes/:3.0.0]
        at com.navercorp.pinpoint.grpc.server.lifecycle.DefaultPingEventHandler.connect(DefaultPingEventHandler.java:59) [pinpoint-grpc-3.0.0.jar:3.0.0]
        at com.navercorp.pinpoint.collector.receiver.grpc.service.AgentService$2.onNext(AgentService.java:106) [classes/:3.0.0]
        at com.navercorp.pinpoint.collector.receiver.grpc.service.AgentService$2.onNext(AgentService.java:94) [classes/:3.0.0]
        at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262) [grpc-stub-1.62.2.jar:1.62.2]
        at io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33) [grpc-api-1.62.2.jar:1.62.2]
        at io.grpc.Contexts$ContextualizedServerCallListener.onMessage(Contexts.java:76) [grpc-api-1.62.2.jar:1.62.2]
        at io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33) [grpc-api-1.62.2.jar:1.62.2]
        at io.grpc.Contexts$ContextualizedServerCallListener.onMessage(Contexts.java:76) [grpc-api-1.62.2.jar:1.62.2]
        at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:329) [grpc-core-1.62.2.jar:1.62.2]
        at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:314) [grpc-core-1.62.2.jar:1.62.2]
        at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:834) [grpc-core-1.62.2.jar:1.62.2]
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) [grpc-core-1.62.2.jar:1.62.2]
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) [grpc-core-1.62.2.jar:1.62.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.net.SocketTimeoutException: callTimeout=10000, callDuration=10433: Failed to get user region lock in10000 ms.  for accessing meta region server. row '***-ami-03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x7F\xFF\xFEk}cx\xB5' on table 'AgentEvent' at null
        at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:157) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.HTable.lambda$put$3(HTable.java:566) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:554) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate$5.doInTable(HbaseTemplate.java:243) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate$5.doInTable(HbaseTemplate.java:240) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate.execute(HbaseTemplate.java:852) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
        ... 47 more
Caused by: org.apache.hadoop.hbase.client.LockTimeoutException: Failed to get user region lock in10000 ms.  for accessing meta region server.
        at org.apache.hadoop.hbase.client.ConnectionImplementation.takeUserRegionLock(ConnectionImplementation.java:1133) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:996) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:897) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.HRegionLocator.lambda$getRegionLocation$0(HRegionLocator.java:80) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.HRegionLocator.tracedLocationFuture(HRegionLocator.java:162) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:79) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.RegionLocator.getRegionLocation(RegionLocator.java:71) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.RegionLocator.getRegionLocation(RegionLocator.java:60) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:219) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:101) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.HTable.lambda$put$3(HTable.java:566) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:554) ~[hbase-shaded-client-2.5.5-hadoop3.jar:2.5.5-hadoop3]
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate$5.doInTable(HbaseTemplate.java:243) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate$5.doInTable(HbaseTemplate.java:240) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
        at com.navercorp.pinpoint.common.hbase.HbaseTemplate.execute(HbaseTemplate.java:852) ~[pinpoint-commons-hbase-3.0.0.jar:3.0.0]
        ... 47 more
@1inuxboy 1inuxboy changed the title Pinpoint 3.0 via Docker Compose How can we troubleshoot OutOfMemoryError in Pinpoint 3.0 Collector Component? Jan 20, 2025
@emeroad
Copy link
Member

emeroad commented Jan 20, 2025

HBASE in a docker environment is not intended for production.

If you want to use it in production, deploy the fully distributed hbase cluster.

@1inuxboy
Copy link
Author

HBASE in a docker environment is not intended for production.

If you want to use it in production, deploy the fully distributed hbase cluster.

Thank you for your reply. If possible, I hope you can provide documentation for deploying the core components of the latest version in a fully distributed deployment while integrating Docker-deployed components. Alternatively, you can provide a deployment concept, and I will try it out. This way, it can help others quickly complete a full-featured production setup of Pinpoint using Docker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@emeroad @1inuxboy