Skip to content
Snippets Groups Projects

Fixing Importer Errors

Last edited by paulwalk

Rake task to fix relationships

  • Start a console in docker
  • Get the path for the importer csv file in tmp/imports (for example "tmp/imports/122_20221017111421/folder_data.csv"

Run the rake task

rdms:create_relationships["tmp/imports/122_20221017111421/folder_data.csv"]

This takes a long while to complete. I ran this on A07 import and about an hour has gone and it's still running.

Note: Start a rails console on the web docker container (see last section on how to do this)

To fix missing permissions (with works not visible)

Fix for all groups

Collection.where(crc_collection_type: "group").each do |collection|
  template = Hyrax::PermissionTemplate.find_by!(source_id: collection.id)

  collection.child_works.each do |work|
    Hyrax::PermissionTemplateApplicator.apply(template).to(model: work)

    work.save
  end
end

Fix for a specific group

group_id = 'x920fw84d'

collection = Collection.find(group_id)
template = Hyrax::PermissionTemplate.find_by!(source_id: collection.id)

collection.child_works.each do |work|
  Hyrax::PermissionTemplateApplicator.apply(template).to(model: work)
  work.save
end

To run failed work job from the rails console 

i = Bulkrax::ImportWorkJob.new
entry_id = 11
importer_id = 1
i.perform(entry_id, importer_id)

# If it passes
::Bulkrax::ImporterRun.find(importer_id).decrement!(:failed_works)

# Sometimes, we have to decrement the failed count if it fails again, as it increases the failed count, but the job has already been counted as failed.

To run failed fileset job from the rails console 

i = Bulkrax::ImportFileSetJob.new
entry_id = 15
importer_id = 1
i.perform(entry_id, importer_id)

# If it passes
::Bulkrax::ImporterRun.find(importer_id).decrement!(:failed_file_sets)

# Sometimes, we have to decrement the failed count if it fails again, as it increases the failed count, but the job has already been counted as failed.

To increase number on works :

begin
    ::Bulkrax::ImporterRun.find(importer_id).increment!(:processed_records)
    ::Bulkrax::ImporterRun.find(importer_id).increment!(:processed_collections)
    ::Bulkrax::ImporterRun.find(importer_id).decrement!(:enqueued_records) unless ::Bulkrax::ImporterRun.find(importer_id).enqueued_records <= 0
rescue => e
    ::Bulkrax::ImporterRun.find(importer_id).increment!(:failed_records)
    ::Bulkrax::ImporterRun.find(importer_id).increment!(:failed_collections)
    ::Bulkrax::ImporterRun.find(importer_id).decrement!(:enqueued_records) unless 
end

The reason why these counts are important, is because the scheduled relationship job will not run if any of the work, fileset or collection jobs are pending. Sometimes, all of the jobs finish successfully, but the importer has got messed up with the count and so will keep waiting.

To record status on entry

entry = Entry.find(entry_id)
entry.importer.current_run = ::Bulkrax::ImporterRun.find(importer_id)
entry.importer.record_status

To run failed collection job from the rails console

i = Bulkrax::ImportCollectionJob.new
entry = 15
importer = 1
i.perform(entry, importer)

To run scheduled relation job from the rails console 

i = Bulkrax::ScheduleRelationshipsJob.new
importer_id = 1
i.perform(importer_id)

This will only run if the job has no pending jobs

Otherwise, to run the scheduler

importer = Importer.find(importer_id)
mporter.last_run.parents.each do |parent_id|
        CreateRelationshipsJob.perform_later(parent_identifier: parent_id, importer_run_id: importer.last_run.id)
      end

To get to the rails console in the web container

Get name of container from docker

$ docker ps
CONTAINER ID   IMAGE               COMMAND                  CREATED      STATUS        PORTS                                       NAMES
0f656337c0e1   rdms-web            "bash -c /bin/docker…"   3 days ago   Up 37 hours   0.0.0.0:3000->3000/tcp, :::3000->3000/tcp   rdms-web-1

Run a shell in the container

docker exec -it rdms-web-1 /bin/bash

Start the rails console

rails c

Comments

    Please register or sign in to add a comment.