Unable to retry jobs that have limits_concurrency is mission control is on a separate rails instance

[salmonsteak1]

I believe this is similar to #200, where there is a reference to the job_class within the mission control instance. This happens when we're running mission control on a separate instance from our main server and it does not contain the job classes needed to perform retries on failed jobs that has the limits_concurrency keyword

[rosa]

@salmonsteak1, could you copy the error trace you get?

[salmonsteak1]

Yeah, here's the trace

[709535af-531d-4bda-9a2d-41eaf054e5d0] activerecord (7.1.5) lib/active_record/locking/pessimistic.rb:95:in `with_lock'                                                                                                                            │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] solid_queue (1.0.1) app/models/solid_queue/failed_execution.rb:23:in `block in retry'                                                                                                                      │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] activesupport (7.1.5) lib/active_support/notifications.rb:206:in `block in instrument'                                                                                                                     │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] activesupport (7.1.5) lib/active_support/notifications/instrumenter.rb:58:in `instrument'                                                                                                                  │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] activesupport (7.1.5) lib/active_support/notifications.rb:206:in `instrument'                                                                                                                              │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] solid_queue (1.0.1) lib/solid_queue.rb:67:in `instrument'                                                                                                                                                  │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] solid_queue (1.0.1) app/models/solid_queue/failed_execution.rb:22:in `retry'                                                                                                                               │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] solid_queue (1.0.1) app/models/solid_queue/job/retryable.rb:15:in `retry'                                                                                                                                  │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] mission_control-jobs (1.0.1) lib/active_job/queue_adapters/solid_queue_ext.rb:61:in `retry_job'                                                                                                            │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] mission_control-jobs (1.0.1) lib/active_job/jobs_relation.rb:144:in `retry_job'                                                                                                                            │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] mission_control-jobs (1.0.1) lib/active_job/executing.rb:20:in `retry'                                                                                                                                     │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] mission_control-jobs (1.0.1) app/controllers/mission_control/jobs/retries_controller.rb:5:in `create'                                                                                                      │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'                                                                                                                 │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/abstract_controller/base.rb:224:in `process_action'                                                                                                                                 │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_controller/metal/rendering.rb:165:in `process_action'                                                                                                                        │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/abstract_controller/callbacks.rb:259:in `block in process_action'                                                                                                                   │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] activesupport (7.1.5) lib/active_support/callbacks.rb:121:in `block in run_callbacks'                                                                                                                      │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] i18n (1.14.6) lib/i18n.rb:353:in `with_locale'                                                                                                                                                             │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] mission_control-jobs (1.0.1) app/controllers/mission_control/jobs/application_controller.rb:26:in `set_current_locale'                                                                                     │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] activesupport (7.1.5) lib/active_support/callbacks.rb:130:in `block in run_callbacks'                                                                                                                      │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] mission_control-jobs (1.0.1) lib/mission_control/jobs/adapter.rb:3:in `activating'                                                                                                                         │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] mission_control-jobs (1.0.1) lib/mission_control/jobs/server.rb:19:in `activating'                                                                                                                         │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] mission_control-jobs (1.0.1) app/controllers/concerns/mission_control/jobs/application_scoped.rb:28:in `activating_job_server'                                                                             │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] activesupport (7.1.5) lib/active_support/callbacks.rb:130:in `block in run_callbacks'                                                                                                                      │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] turbo-rails (2.0.11) lib/turbo-rails.rb:24:in `with_request_id'                                                                                                                                            │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] turbo-rails (2.0.11) app/controllers/concerns/turbo/request_id_tracking.rb:10:in `turbo_tracking_request_id'                                                                                               │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] activesupport (7.1.5) lib/active_support/callbacks.rb:130:in `block in run_callbacks'                                                                                                                      │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actiontext (7.1.5) lib/action_text/rendering.rb:23:in `with_renderer'                                                                                                                                      │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actiontext (7.1.5) lib/action_text/engine.rb:69:in `block (4 levels) in <class:Engine>'
[709535af-531d-4bda-9a2d-41eaf054e5d0] activesupport (7.1.5) lib/active_support/callbacks.rb:130:in `instance_exec'                                                                                                                               │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] activesupport (7.1.5) lib/active_support/callbacks.rb:130:in `block in run_callbacks'                                                                                                                      │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] activesupport (7.1.5) lib/active_support/callbacks.rb:141:in `run_callbacks'                                                                                                                               │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/abstract_controller/callbacks.rb:258:in `process_action'                                                                                                                            │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_controller/metal/rescue.rb:25:in `process_action'                                                                                                                            │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_controller/metal/instrumentation.rb:74:in `block in process_action'                                                                                                          │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] activesupport (7.1.5) lib/active_support/notifications.rb:206:in `block in instrument'                                                                                                                     │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] activesupport (7.1.5) lib/active_support/notifications/instrumenter.rb:58:in `instrument'                                                                                                                  │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] activesupport (7.1.5) lib/active_support/notifications.rb:206:in `instrument'                                                                                                                              │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_controller/metal/instrumentation.rb:73:in `process_action'                                                                                                                   │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_controller/metal/params_wrapper.rb:261:in `process_action'                                                                                                                   │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] activerecord (7.1.5) lib/active_record/railties/controller_runtime.rb:32:in `process_action'                                                                                                               │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/abstract_controller/base.rb:160:in `process'                                                                                                                                        │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionview (7.1.5) lib/action_view/rendering.rb:40:in `process'                                                                                                                                            │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_controller/metal.rb:227:in `dispatch'                                                                                                                                        │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_controller/metal.rb:309:in `dispatch'                                                                                                                                        │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'                                                                                                                               │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/routing/route_set.rb:32:in `serve'                                                                                                                                  │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/journey/router.rb:51:in `block in serve'                                                                                                                            │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/journey/router.rb:131:in `block in find_routes'                                                                                                                     │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/journey/router.rb:124:in `each'                                                                                                                                     │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/journey/router.rb:124:in `find_routes'                                                                                                                              │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/journey/router.rb:32:in `serve'                                                                                                                                     │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/routing/route_set.rb:882:in `call'                                                                                                                                  │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] railties (7.1.5) lib/rails/engine.rb:536:in `call'                                                                                                                                                         │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] railties (7.1.5) lib/rails/railtie.rb:226:in `public_send'                                                                                                                                                 │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] railties (7.1.5) lib/rails/railtie.rb:226:in `method_missing'                                                                                                                                              │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/routing/mapper.rb:31:in `block in <class:Constraints>'                                                                                                              │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/routing/mapper.rb:60:in `serve'                                                                                                                                     │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/journey/router.rb:51:in `block in serve'                                                                                                                            │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/journey/router.rb:131:in `block in find_routes'                                                                                                                     │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/journey/router.rb:124:in `each'                                                                                                                                     │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/journey/router.rb:124:in `find_routes'                                                                                                                              │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/journey/router.rb:32:in `serve'                                                                                                                                     │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/routing/route_set.rb:882:in `call'                                                                                                                                  │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] rack (3.1.8) lib/rack/tempfile_reaper.rb:20:in `call'                                                                                                                                                      │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] rack (3.1.8) lib/rack/etag.rb:29:in `call'                                                                                                                                                                 │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] rack (3.1.8) lib/rack/conditional_get.rb:43:in `call'                                                                                                                                                      │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] rack (3.1.8) lib/rack/head.rb:15:in `call'                                                                                                                                                                 │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/http/permissions_policy.rb:36:in `call'                                                                                                                             │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/http/content_security_policy.rb:33:in `call'                                                                                                                        │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] rack-session (2.0.0) lib/rack/session/abstract/id.rb:272:in `context'                                                                                                                                      │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] rack-session (2.0.0) lib/rack/session/abstract/id.rb:266:in `call'                                                                                                                                         │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/middleware/cookies.rb:689:in `call'                                                                                                                                 │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
|[709535af-531d-4bda-9a2d-41eaf054e5d0] activesupport (7.1.5) lib/active_support/callbacks.rb:101:in `run_callbacks'                                                                                                                               │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/middleware/callbacks.rb:28:in `call'                                                                                                                                │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'                                                                                                                         │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'                                                                                                                          │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] railties (7.1.5) lib/rails/rack/logger.rb:37:in `call_app'                                                                                                                                                 │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] railties (7.1.5) lib/rails/rack/logger.rb:24:in `block in call'                                                                                                                                            │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] activesupport (7.1.5) lib/active_support/tagged_logging.rb:139:in `block in tagged'                                                                                                                        │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] activesupport (7.1.5) lib/active_support/tagged_logging.rb:39:in `tagged'                                                                                                                                  │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] activesupport (7.1.5) lib/active_support/tagged_logging.rb:139:in `tagged'                                                                                                                                 │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] activesupport (7.1.5) lib/active_support/broadcast_logger.rb:241:in `method_missing'                                                                                                                       │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] railties (7.1.5) lib/rails/rack/logger.rb:24:in `call'                                                                                                                                                     │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/middleware/remote_ip.rb:92:in `call'                                                                                                                                │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/middleware/request_id.rb:28:in `call'                                                                                                                               │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] rack (3.1.8) lib/rack/method_override.rb:28:in `call'                                                                                                                                                      │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] rack (3.1.8) lib/rack/runtime.rb:24:in `call'                                                                                                                                                              │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/middleware/executor.rb:14:in `call'                                                                                                                                 │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] actionpack (7.1.5) lib/action_dispatch/middleware/static.rb:25:in `call'                                                                                                                                   │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] rack (3.1.8) lib/rack/sendfile.rb:114:in `call'                                                                                                                                                            │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] railties (7.1.5) lib/rails/engine.rb:536:in `call'                                                                                                                                                         │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] puma (6.4.3) lib/puma/configuration.rb:272:in `call'                                                                                                                                                       │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] puma (6.4.3) lib/puma/request.rb:100:in `block in handle_request'                                                                                                                                          │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] puma (6.4.3) lib/puma/thread_pool.rb:378:in `with_force_shutdown'                                                                                                                                          │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] puma (6.4.3) lib/puma/request.rb:99:in `handle_request'                                                                                                                                                    │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] puma (6.4.3) lib/puma/server.rb:464:in `process_client'                                                                                                                                                    │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] puma (6.4.3) lib/puma/server.rb:245:in `block in run'                                                                                                                                                      │
│ [709535af-531d-4bda-9a2d-41eaf054e5d0] puma (6.4.3) lib/puma/thread_pool.rb:155:in `block in spawn_thread'

[rosa]

Thank you! Is the error happening in the call to with_lock? That's very strange 😕 Is that the full trace? Could you copy the error as well?

[salmonsteak1]

@rosa ah okay, I managed to find more logs:

Here's also the error that was thrown:

Module::DelegationError (SolidQueue::Job#concurrency_duration delegated to job_class.concurrency_duration, but job_class is nil: #<SolidQueue::Job id: 123, queue_name: "xxx", class_name: "xxxJob", arguments: