add query to remove user ids from filepaths

[pahatz]

Related issue(s) and PR(s)

This PR closes #1123.

Description

Previous work from #1097 removed the user ids from the filepaths. This sql query now should take care of the current databases and update to remove the user ids from the filepaths there as well.

How to test

It has been tested manually with both test environment and production data.

bring up the test env

make-sda-s3-up

fill 4 test entries in the database

query=$(cat <<-END
    INSERT INTO sda.files (submission_user, submission_file_path)
VALUES
	('[email protected]', 'test_lifescience-ri.eu/dataset_test899898/dataset/IMAGES/IMAGE_test/1.2.826.0.1.00000.8.498.000.dcm.c4gh'),
	('testtesttest', 'testtesttest/dataset_krtkl45/picostorage/default/output/encrypted/Image_klfmgrY/1.2.000.0.1.3680000.8.111.222.dcm.c4gh'),
	('[email protected]', '[email protected]/dataset_dd33rr/media/lala/default/output/encrypted/lala_lolo/1.2.826.0.1.3333.8.498.1111.dcm.c4gh'),
	('[email protected]', '[email protected]/wrongname/picostorage/default/output/encrypted/lala/1.2.826.0.1.11.22.dcm.c4gh');
END

)

docker exec -it postgres psql -U postgres -d sda -c "$query"

run the script

docker cp postgresql/migratedb.d/17.sql postgres:/17.sql
docker exec -it postgres psql -U postgres -d sda -f /17.sql

check data is right

docker exec -it postgres psql -U postgres -d sda -c "select submission_user, submission_file_path from sda.files"

the user ids should be gone from the submission_file_path

[jbygdell]

This solution is a bit harsh since it will remove the first part of the path even if is not a username.

Suggested change

	SET submission_file_path = regexp_replace(submission_file_path, '^[^/]*/', '');
	SET submission_file_path = REPLACE(submission_file_path, CONCAT(REPLACE(submission_user, '@', '_'),'/'), '');;

[pahatz]

The reason I went this way is that we have some old entries in BP that are missing the @ domain suffix in users and the corresponding _domain suffix in file path.
But the following seems to work for those cases too:

UPDATE sda.files
SET submission_file_path = CASE
	WHEN LENGTH(submission_file_path) = 40 THEN regexp_replace(submission_file_path, '[@_]/^[^/]', '')
	ELSE REPLACE(submission_file_path, CONCAT(REPLACE(submission_user, '@', '_'),'/'), '')
END;

[jbygdell]

I think this maybe sholdn't even be a .sql file by´ut rather a readme so the DB admin knows that it needs to be done. Because this is not something that should be allowed to happen automatically.

[pahatz]

I agree, let's discuss this on a daily.

[MalinAhlberg]

Just a note, #1161 was merged into feature/do-not-leak-user-ids, so that all related work could be grouped there before merged to main. Maybe this one should go there to? Or should feature/do-not-leak-user-ids be merged after this one?