Having Problems configuring solid_queue DB

[Kenneth-joseph]

Apologies if this question has already been asked but I can't find the right way to set up the solid_queue db alongside my primary db. Every time I try to start tsolid_queue I keep getting the error below, I'm all new when it comes to using multiple DBs and I'm here to learn:

bundle exec rake solid_queue:start rake aborted! ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR: relation "solid_queue_processes" does not exist LINE 10: WHERE a.attrelid = '"solid_queue_processes"'::regclass

This is my database.yml file setup:

`default: &default
adapter: postgresql
pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
timeout: 5000

development:
primary:
<<: *default
database: msangia_development
queue:
<<: *default
database: msangia_queue_dev
migrations_paths: db/queue_migrate
test:
<<: *default
database: msangia_wifi_test

production:
primary:
<<: *default
database: msangia_wifi_production
queue:
<<: *default
database: app_production_queue
migrations_paths: db/queue_migrate`

development.rb environment:

`config.active_job.queue_adapter = :solid_queue

config.solid_queue.connects_to = { database: { writing: :queue } }`

The queue_schema.rb is created, but every time I run rails:db prepare, it's cleared, and nothing happens after. I did run the migrations on my terminal, but I'm still getting the same error when I try to start solid_queue. I created the queue_migrate, copied the queue_schema into it, and tried migrating it, but I'm still getting the same error.
I would appreciate any help, thanks in prior.

[rosa]

Hey @Kenneth-joseph, sorry for the trouble! Could you let me know which version of Rails you are using?

[DonSchado]

Yes, I had the same issue. (tested with 7.2 and 8.0.0.rc2)

first make sure the schema is really loaded into the db.
In my case I checked with "DB Browser for SQLite" and saw that it's missing all the tables.

bin/rails db:schema:load:queue
# Load a database schema file (either db/schema.rb or db/structure.sql, depending on `ENV['SCHEMA_FORMAT']` or `config.active_record.schema_format`) into the queue database

Then I had to add the storage config also to the test env (in database.yml).

actually rails db:reset:queue gives a hint, when you don't have it configured for test:

❯ bin/rails db:reset:queue
Dropped database 'storage/development_queue.sqlite3'
Created database 'storage/development_queue.sqlite3'
bin/rails aborted!
TypeError: Invalid type for configuration. Expected Symbol, String, or Hash. Got nil (TypeError)

       raise TypeError, "Invalid type for configuration. Expected Symbol, String, or Hash. Got #{config.inspect}"
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Tasks: TOP => db:reset:queue => db:setup:queue => db:schema:load:queue => db:test:purge:queue
(See full trace by running task with --trace)

So this is the complete change I made after rails new:

diff --git a/app/jobs/application_job.rb b/app/jobs/application_job.rb
index d394c3d..a20ed1b 100644
--- a/app/jobs/application_job.rb
+++ b/app/jobs/application_job.rb
@@ -4,4 +4,7 @@ class ApplicationJob < ActiveJob::Base

  # Most jobs are safe to ignore if the underlying records are no longer available
  # discard_on ActiveJob::DeserializationError
+  def perform
+    puts 'ohai!'
+  end
end
diff --git a/config/database.yml b/config/database.yml
index 2640cb5..ed7bd1a 100644
--- a/config/database.yml
+++ b/config/database.yml
@@ -10,15 +10,25 @@ default: &default
  timeout: 5000

development:
-  <<: *default
-  database: storage/development.sqlite3
+  primary:
+    <<: *default
+    database: storage/development.sqlite3
+  queue:
+    <<: *default
+    database: storage/development_queue.sqlite3
+    migrations_paths: db/queue_migrate

# Warning: The database defined as "test" will be erased and
# re-generated from your development database when you run "rake".
# Do not set this db to the same as development or production.
test:
-  <<: *default
-  database: storage/test.sqlite3
+  primary:
+    <<: *default
+    database: storage/test.sqlite3
+  queue:
+    <<: *default
+    database: storage/test_queue.sqlite3
+    migrations_paths: db/queue_migrate


# Store production database in the storage/ directory, which by default
diff --git a/config/environments/development.rb b/config/environments/development.rb
index 4cc21c4..11a03de 100644
--- a/config/environments/development.rb
+++ b/config/environments/development.rb
@@ -69,4 +69,6 @@ Rails.application.configure do

  # Apply autocorrection by RuboCop to files generated by `bin/rails generate`.
  # config.generators.apply_rubocop_autocorrect_after_generate!
+ config.active_job.queue_adapter = :solid_queue
+ config.solid_queue.connects_to = { database: { writing: :queue } }
end
diff --git a/config/environments/test.rb b/config/environments/test.rb
index c2095b1..87e05c2 100644
--- a/config/environments/test.rb
+++ b/config/environments/test.rb
@@ -50,4 +50,6 @@ Rails.application.configure do

  # Raise error when a before_action's only/except options reference missing actions.
  config.action_controller.raise_on_missing_callback_actions = true
+ config.active_job.queue_adapter = :solid_queue
+ config.solid_queue.connects_to = { database: { writing: :queue } }
end

❯ bin/rails runner "ApplicationJob.perform_later"

❯ bin/jobs
ohai!

hope that helps :)

[truesteez]

I'm facing the same issue right now with postgres

[truesteez]

I was able to fix this error by running rails db:prepare in development so I guess that part is really important for solid queue as I didn't see any migration files being added only updates to the schema

[Kenneth-joseph]

Hello @rosa
I'm using Rails 7.2.1
The only way I had to go to make solid_queue start was to comment out the connects_to line in my dev environment.
Commented this out:
# config.solid_queue.connects_to = { database: { writing: :queue } }

and defaulted to using a single database. My only concern is whether this is the ideal approach for production since I wouldn't want my database to be overloaded or clustered.
Any help with setting up the multiple db or further essential insights pointing to the most reasonable approach to take would be highly appreciated.

[kylekeesling]

@indigotechtutorials if you’re upgrading from earlier versions you’ll need to manually step through a few of them to generate the necessary migrations. You can see notes on it here:

https://github.com/rails/solid_queue/blob/main/UPGRADING.md

[sevinchek]

I am using Rails 8 and I have solid_queue working in development.
After creating the project I ran:
bin/rails db:setup
bin/rails db:migrate

and made these configurations:

# config/database.yml
development:
  primary: &primary_development
    <<: *default
    database: athens_development
  cache:
    <<: *primary_development
    database: athens_development_cache
    migrations_paths: db/cache_migrate
  queue:
    <<: *primary_development
    database: athens_development_queue
    migrations_paths: db/queue_migrate
  cable:
    <<: *primary_development
    database: athens_development_cable
    migrations_paths: db/cable_migrate

# config/environments/development.rb
config.active_job.queue_adapter = :solid_queue
config.solid_queue.connects_to = { database: { writing: :queue } }

[dpaluy]

When running rails solid_queue:install, it creates schema file only and doesn't generate the migration. After running commands like: rails db:prepare the schema is reset because the migration is missing.

I solved it by manually creating the migration:
rails generate migration CreateQueueTable --database=queue

It will create a regular migration into db/queue_migrate folder. Copy the content of queue_schema.rb.

class CreateQueueTable < ActiveRecord::Migration[8.0]
  def change
    create_table "solid_queue_blocked_executions", force: :cascade do |t|
      t.bigint "job_id", null: false
      t.string "queue_name", null: false
      t.integer "priority", default: 0, null: false
      t.string "concurrency_key", null: false
      t.datetime "expires_at", null: false
      t.datetime "created_at", null: false
      t.index [ "concurrency_key", "priority", "job_id" ], name: "index_solid_queue_blocked_executions_for_release"
      t.index [ "expires_at", "concurrency_key" ], name: "index_solid_queue_blocked_executions_for_maintenance"
      t.index [ "job_id" ], name: "index_solid_queue_blocked_executions_on_job_id", unique: true
    end

    create_table "solid_queue_claimed_executions", force: :cascade do |t|
      t.bigint "job_id", null: false
      t.bigint "process_id"
      t.datetime "created_at", null: false
      t.index [ "job_id" ], name: "index_solid_queue_claimed_executions_on_job_id", unique: true
      t.index [ "process_id", "job_id" ], name: "index_solid_queue_claimed_executions_on_process_id_and_job_id"
    end
    
    # etc...
 end

Now you can run rails db:prepare and it will work for both development and production.

Note: remember to add database.yml config, as mentioned by @sevinchek

[memorycancel]

Same issue.
First, my env is:

rails 8.0.1
ruby 3.3.5
# create clean rails project
rails new rails-jobs

then I follow the steps of [README](https://github.com/rails/solid_queue?tab=readme-ov-file#usage-in-development-and-other-non-production-environments)

it not work since the queue_migrate is empty, database can not get the tables of solid_queue.

[memorycancel]

In my dev env, I use export RAILS_ENV=production, it works and everything is ok, I guess there something wrong with the development env.
I think maybe the dev and test env is no necessary.

[rosa]

it not work since the queue_migrate is empty, database can not get the tables of solid_queue.

As long as you're running db:prepare, the DB should get populated with the tables because these live in queue_schema.rb. You don't need migrations.

[memorycancel]

it not work since the queue_migrate is empty, database can not get the tables of solid_queue.

As long as you're running db:prepare, the DB should get populated with the tables because these live in queue_schema.rb. You don't need migrations.

I see, thanks, I confused with the command of bin/rails db:migrate:queue here: https://edgeguides.rubyonrails.org/active_job_basics.html#development

[rosa]

Oh, I see, that's wrong. The instructions here are the correct ones. I'll open a PR in the Rails guides to fix this.

[rosa]

Done in rails/rails#54101

[memorycancel]

Done in rails/rails#54101

LGTM.

Additionly, seems that must run the command bin/rails solid_queue:install then generate the bin/jobs, but the README says this is default in Rails 8. @rosa

[rosa]

@memorycancel hmm... I think in Rails 8 when you generate a new app, solid_queue:install is run automatically unless explicitly skipped.

[memorycancel]

@memorycancel hmm... I think in Rails 8 when you generate a new app, solid_queue:install is run automatically unless explicitly skipped.

I found it works on my MacBook(MacOS) but not work on Laptop (Ubuntu Desktop 24.04) . The version of Rails is same(8.0.1)
since I type ^C before run bundle install --quiet DONE when I create the new proj(rails new)