getting Paperless working with locally delivered mail attachements? #17

xpseudonym · 2024-02-06T15:48:10Z

xpseudonym
Feb 6, 2024

First up - thanks @jdoss for the script that's had me up and running.

I've been going through the motions of getting this working with local email delivery - my rather old Ricoh Aficio 1515MF is very good at keep'in on going, but, connectivity has always been a problem - no ftp (let alone sftp) and no secure email - but, insecure email - meaning I can not connect to any public email service these days - but, I can connect to all my local machines running postfix etc.

Paperless doesn't support any local mail delivery options, :-/
paperless-ngx/paperless-ngx#4538 (comment)
So, I set about sorting it out for myself. I figured getting local mail delivery set up for the ppngx user (paperless) and putting the following in .forward should provide what I want:

"|/usr/bin/ripmime -i - --no-nameless -d ~/.local/share/containers/storage/volumes/paperless-consume/_data"

But, no dice - I can get delivery to ~/ but not to the consume volume. I smelt SELinux and found:
https://blog.christophersmart.com/2021/01/31/podman-volumes-and-selinux/

So, two questions:
1/ re the above link, the declaration for the consume directory already has a little :z at the end of it:

-v paperless-consume:/usr/src/paperless/consume:U,z \

So, I'm unsure what more is needed? Any pointers on how to get this working?
2/ I have no need of SFTPGo - having replaced that function with locally delivered mail attachments - if I remove all references to it in start.sh will that stuff other things up...?

Thanks

Answered by xpseudonym

Feb 18, 2024

Right then!
Success!!!
Of sorts...
Doing some more reading - it seemed --user root shouldn't be needed - as that's default - so, first attempt with changing -e USERMAP_UID=0 & -e USERMAP_GID=0 only - and that worked - after turning off SELinux...

So, next todo according to sealert:

# ausearch -c 'ripmime' --raw | audit2allow -M my-ripmime
# semodule -X 300 -i my-ripmime.pp

Done! (For now.)
I'm unsure whether it would be a better to relabel the consume and export directories rather than adding an exception?:

[paperless@server ppngx]$ ls -lhZ
total 32K
-rw-r--r--. 1 paperless paperless unconfined_u:object_r:user_home_t:s0            3.3K Feb  9 20:24 backup.sh
drwxr-xr-x. 2 paperless paper…

View full answer

jdoss · 2024-02-06T18:06:04Z

jdoss
Feb 6, 2024
Maintainer

First up - thanks @jdoss for the script that's had me up and running.

Wonderful! You are welcome. :)

I've been going through the motions of getting this working with local email delivery - my rather old Ricoh Aficio 1515MF is very good at keep'in on going, but, connectivity has always been a problem - no ftp (let alone sftp) and no secure email - but, insecure email - meaning I can not connect to any public email service these days - but, I can connect to all my local machines running postfix etc.

Paperless doesn't support any local mail delivery options, :-/ paperless-ngx/paperless-ngx#4538 (comment) So, I set about sorting it out for myself. I figured getting local mail delivery set up for the ppngx user (paperless) and putting the following in .forward should provide what I want:
"|/usr/bin/ripmime -i - --no-nameless -d ~/.local/share/containers/storage/volumes/paperless-consume/_data"

Neat. This the first I have heard of ripmime.

So, two questions: 1/ re the above link, the declaration for the consume directory already has a little :z at the end of it:
-v paperless-consume:/usr/src/paperless/consume:U,z \

Since you are not using SFTPGo you can change this mount to a local path on the disk like this:

Make your consume directory (can be where ever you want)

mkdir -p ${HOME}/Documents/paperless/consume

Edit start.sh and change

-v paperless-consume:/usr/src/paperless/consume:U,z \

to

-v ${HOME}/Documents/paperless/consume:/usr/src/paperless/consume:z \

and then you can set your .forward to:

"|/usr/bin/ripmime -i - --no-nameless -d ${HOME}/Documents/paperless/consume"

and I think you will be business. There can be some user permissions issues with how the files are written by your user via ripmime and the user inside the Paperless container. Test the above out and see. I think it will be fine since the Paperless container runs as the root user inside the container:

$ podman exec -it paperless-webserver bash
root@paperless:/usr/src/paperless/src# id
uid=0(root) gid=0(root) groups=0(root)

and rootless Podman should handle the permissions, I think.

So, I'm unsure what more is needed? Any pointers on how to get this working? 2/ I have no need of SFTPGo - having replaced that function with locally delivered mail attachments - if I remove all references to it in start.sh will that stuff other things up...?

Yah just remove it from the script. It should work well.

Let me know how it turns out or if you have any issues and I'd be glad to help.

0 replies

xpseudonym · 2024-02-06T20:33:52Z

xpseudonym
Feb 6, 2024
Author

Thanks - I'll be back...
btw, took me ages of thrashing about with mail before discovering ripmime - and had to get it built just for me...:
https://bugzilla.redhat.com/show_bug.cgi?id=2262664
(I could have built it myself of course, but then that's no fun for anyone else.)

0 replies

xpseudonym · 2024-02-09T21:40:29Z

xpseudonym
Feb 9, 2024
Author

Hmm, @jdoss I got stuck...
Here's what I did:

#!/bin/bash

...

echo "Starting Paperless..."
podman create --replace --pod paperless \
  --name paperless-webserver \
  --restart=unless-stopped \
  --stop-timeout=90 \
  --health-cmd='["curl", "-f", "http://localhost:8000"]' \
  --health-retries=5 \
  --health-start-period=60s \
  --health-timeout=10s \
  -e PAPERLESS_REDIS=redis://localhost:${REDIS_PORT} \
  -e PAPERLESS_DBHOST=localhost \
  -e PAPERLESS_DBNAME=${POSTGRES_USER} \
  -e PAPERLESS_DBPASS=${POSTGRESQL_PASSWORD} \
  -e PAPERLESS_TIKA_ENABLED=1 \
  -e PAPERLESS_TIKA_GOTENBERG_ENDPOINT=http://localhost:3000 \
  -e PAPERLESS_TIKA_ENDPOINT=http://localhost:9998 \
  -e PAPERLESS_URL=${PAPERLESS_URL} \
  -e USERMAP_UID=${PAPERLESS_UID} \
  -e USERMAP_GID=${PAPERLESS_GID} \
  -e PAPERLESS_SECRET_KEY=${PAPERLESS_SECRET_KEY} \
  -e PAPERLESS_TIME_ZONE=${PAPERLESS_TIME_ZONE} \
  -e PAPERLESS_OCR_LANGUAGE=${PAPERLESS_OCR_LANGUAGE} \
  -v paperless-data:/usr/src/paperless/data:Z \
  -v paperless-media:/usr/src/paperless/media:Z \
#  -v paperless-consume:/usr/src/paperless/consume:U,z \
  -v ${HOME}/consume:/usr/src/paperless/consume:z \
  -v ${PWD}/export:/usr/src/paperless/export:U,Z \
  ghcr.io/paperless-ngx/paperless-ngx:${PAPERLESS_VERSION}
podman start paperless-webserver

...

And, this is what I got...:

[paperless@server ppngx]$ ./start.sh
Creating Paperless Pod...
6c068625b83be52788d6ceec36a4b705990abf36cbf465b1d993fa7fe48ceada
Starting Redis...
Trying to pull docker.io/library/redis:6...
Getting image source signatures
Copying blob 0a0eb0ff91b9 done  

...

Copying config b5eecd9657 done  
Writing manifest to image destination
005d45a49ebf594be16e02fd40d6f19d80221d1978ded97caaaef46cec98cb6e
paperless-tika
Starting Paperless...
Error: requires at least 1 arg(s), only received 0
[paperless@server ppngx]$

Is there something I'm missing?

0 replies

jdoss · 2024-02-10T16:42:52Z

jdoss
Feb 10, 2024
Maintainer

Don't comment # -v paperless-consume:/usr/src/paperless/consume:U,z \ remove it.

0 replies

xpseudonym · 2024-02-13T13:59:00Z

xpseudonym
Feb 13, 2024
Author

Hmm, yeah, oops.
I did some reading: How to put a line comment for a multi-line command
So, I haven't commented out another line of script, I've added a comment in the middle of a (wrapped) command line... and everything after # is ignored - including the image to pull - until the next command, podman start paperless-webserver :-/
And, the best I can do is:

...
  -v paperless-media:/usr/src/paperless/media:Z \
  -v ${HOME}/consume:/usr/src/paperless/consume:z \
  -v ${PWD}/export:/usr/src/paperless/export:U,Z \
  ghcr.io/paperless-ngx/paperless-ngx:${PAPERLESS_VERSION}
# swapping '-v ${HOME}/consume:/usr/src/paperless/consume:z \' in the above
# for '-v paperless-consume:/usr/src/paperless/consume:U,z \' in the original.

0 replies

xpseudonym · 2024-02-13T14:22:52Z

xpseudonym
Feb 13, 2024
Author

Some supplementary Qs:

What are the consume and export directories included in the git repo - I was thinking of reusing the consume directory instead of adding more clutter;
redis, postgres and gotenberg are hard coded to versions 6, 13 and 7 - presumably because paperless is - is there any way to track the versions paperless depends on?

1 reply

jdoss Feb 13, 2024
Maintainer

Some supplementary Qs:

1. What are the `consume` and `export` directories included in the git repo - I was thinking of reusing the `consume` directory instead of adding more clutter;

Export is used if you want to export all of your data from Paperless IIRC. I'd keep the separate. Paperless is watching the consume directory and as soon as it is sees changes it will try to import PDFs.

2. `redis`, `postgres` and `gotenberg` are hard coded to versions 6, 13 and 7 - presumably because `paperless` is - is there any way to track the versions `paperless` depends on?

Yeah, the were whatever version they had set in the docker-compose.yaml file in the Paperless repo when I translated the stack to this script. I haven't looked if they need to be updated in a bit. Could be a good first PR to the repo if the versions need to be bumped. 😉

xpseudonym · 2024-02-15T20:52:45Z

xpseudonym
Feb 15, 2024
Author

Some supplementary Qs:
1. What are the `consume` and `export` directories included in the git repo - I was thinking of reusing the `consume` directory instead of adding more clutter;
Export is used if you want to export all of your data from Paperless IIRC. I'd keep the separate. Paperless is watching the consume directory and as soon as it is sees changes it will try to import PDFs.

This is a bit frustrating...
I ran the attached start.sh.txt with:

...
  -v ${PWD}/consume:/usr/src/paperless/consume:z \
  -v ${PWD}/export:/usr/src/paperless/export:U,Z \
...

And, I can't post to ${PWD}/consume. The ownership of both the consume and export directories are 297617 which translates to 999 (if I've got my math right). All of the other container directories seemed okay and to be owned by my user paperless (1010). I changed PAPERLESS_UID and PAPERLESS_GID from the default 1000 to 1010 in the attached, but that made no difference. I did some reading here, but couldn't work it out. Any idea why the ownership of consume and export is set to 999 and how do I change that so it's like the rest of the container volumes and owned by my user paperless (1010)?

[paperless@server ppngx]$ cat /etc/subuid
admin:100000:65536
paperless:296608:65536
[paperless@server ppngx]$ podman unshare cat /proc/self/uid_map
         0       1010          1
         1     296608      65536
[paperless@server ppngx]$ ls -lZ
total 32
-rw-r--r--. 1 paperless paperless unconfined_u:object_r:user_home_t:s0            3315 Feb  9 20:24 backup.sh
drwxr-xr-x. 2    297617    297617 system_u:object_r:container_file_t:s0             24 Feb 15 19:43 consume
drwxr-xr-x. 2    297617    297617 system_u:object_r:container_file_t:s0:c665,c908   24 Feb  9 20:24 export
-rw-r--r--. 1 paperless paperless unconfined_u:object_r:user_home_t:s0            1065 Feb  9 20:24 LICENSE
-rw-r--r--. 1 paperless paperless unconfined_u:object_r:user_home_t:s0            6313 Feb  9 20:24 README.md
-rwxr-xr-x. 1 paperless paperless unconfined_u:object_r:user_home_t:s0            5660 Feb 15 19:12 start.sh
-rwxr-xr-x. 1 paperless paperless unconfined_u:object_r:user_home_t:s0            4947 Feb  9 20:24 start.sh.bak
[paperless@server ppngx]$ cd /var/home/paperless/.local/share/containers/storage/volumes
[paperless@server volumes]$ ls -lZ
total 0
drwx------. 3 paperless paperless unconfined_u:object_r:data_home_t:s0 27 Feb 15 19:37 paperless-data
drwx------. 3 paperless paperless unconfined_u:object_r:data_home_t:s0 19 Feb 15 19:37 paperless-media
drwx------. 3 paperless paperless unconfined_u:object_r:data_home_t:s0 19 Feb 15 19:29 paperless-postgresql
drwx------. 3 paperless paperless unconfined_u:object_r:data_home_t:s0 19 Feb 15 19:29 paperless-redis
[paperless@server volumes]$

4 replies

xpseudonym Feb 16, 2024
Author

I guess an alternative would be to make consume writeable by it's group and add paperless user to that group - but, pretty ugly. Also, re the version numbers - I'd like to work out how to pull those direct from the Paperless docker-compose.postgres-tika.yml here.

jdoss Feb 16, 2024
Maintainer

Eh, I don't wan to make the script too complicated by having it parse YAML if that is what you are thinking about here. Just bumping the versions in the script is good enough.

jdoss Feb 16, 2024
Maintainer

And, I can't post to ${PWD}/consume. The ownership of both the consume and export directories are 297617 which translates to 999 (if I've got my math right). All of the other container directories seemed okay and to be owned by my user paperless (1010). I changed PAPERLESS_UID and PAPERLESS_GID from the default 1000 to 1010 in the attached, but that made no difference. I did some reading here, but couldn't work it out. Any idea why the ownership of consume and export is set to 999 and how do I change that so it's like the rest of the container volumes and owned by my user paperless (1010)?

Welcome to Rootless Podman fun. You can play around with https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md#etcsubuid-and-etcsubgid-configuration to match up the UID and GID to 1010 or you can try running paperless-webserver as the root user inside the container.

To do that, stop paperless, use sudo to nuke ${PWD}/consume and then mkdir ${PWD}/consume to recreate it with the right UID/GID of your user. Then make these changes.

  --user root \
  -e USERMAP_UID=0 \
  -e USERMAP_GID=0 \

In full:

echo "Starting Paperless..."
podman create --replace --pod paperless \
  --name paperless-webserver \
  --restart=unless-stopped \
  --stop-timeout=90 \
  --health-cmd='["curl", "-f", "http://localhost:8000"]' \
  --health-retries=5 \
  --health-start-period=60s \
  --health-timeout=10s \
  --user root \
  -e PAPERLESS_REDIS=redis://localhost:${REDIS_PORT} \
  -e PAPERLESS_DBHOST=localhost \
  -e PAPERLESS_DBNAME=${POSTGRES_USER} \
  -e PAPERLESS_DBPASS=${POSTGRESQL_PASSWORD} \
  -e PAPERLESS_TIKA_ENABLED=1 \
  -e PAPERLESS_TIKA_GOTENBERG_ENDPOINT=http://localhost:3000 \
  -e PAPERLESS_TIKA_ENDPOINT=http://localhost:9998 \
  -e PAPERLESS_URL=${PAPERLESS_URL} \
  -e USERMAP_UID=0 \
  -e USERMAP_GID=0 \
  -e PAPERLESS_SECRET_KEY=${PAPERLESS_SECRET_KEY} \
  -e PAPERLESS_TIME_ZONE=${PAPERLESS_TIME_ZONE} \
  -e PAPERLESS_OCR_LANGUAGE=${PAPERLESS_OCR_LANGUAGE} \
  -v paperless-data:/usr/src/paperless/data:Z \
  -v paperless-media:/usr/src/paperless/media:Z \
  -v ${PWD}/consume:/usr/src/paperless/consume:z \
  -v ${PWD}/export:/usr/src/paperless/export:U,Z \
  ghcr.io/paperless-ngx/paperless-ngx:${PAPERLESS_VERSION}
# swapping out '-v paperless-consume:/usr/src/paperless/consume:U,z \' in the original
# for '-v ${PWD}/consume:/usr/src/paperless/consume:z \' in the above.
podman start paperless-webserver

That should translate the root user to your UID/GID, I think. Good luck.

xpseudonym Feb 17, 2024
Author

To do that, stop paperless, use sudo to nuke ${PWD}/consume and then mkdir ${PWD}/consume to recreate it with the right UID/GID of your user.

Thanks @jdoss for the links to guru Walsh's article.
Re the above, isn't the issue more:
why does Paperless make ${PWD}/consume (and ${PWD}/export) have UID 999, rather than;
why does Podman transpose 999 to 297617?
All the other volume/ directories Paperless makes are okay - that is, UID 1010. Re your original reply (long time ago now...):

... the Paperless container runs as the root user inside the container

That seems correct and gives the paperless user access to all the other container volumes/ directories with UID 1010 (e.g. paperless-data and paperless-media) but, Paperless is giving ${PWD}/consume and ${PWD}/export UID 999.

Between my Q and your A, I did nuke the whole install and recreated ${PWD}/consume and ${PWD}/export with UID 1010 - then, running the script, they got changed back to UID 297617 - grrr. I haven't tried changes you suggest above to podman create yet - so perhaps I'm getting ahead of myself and should go away and come back when I have...

Thx

xpseudonym · 2024-02-18T19:36:42Z

xpseudonym
Feb 18, 2024
Author

Right then!
Success!!!
Of sorts...
Doing some more reading - it seemed --user root shouldn't be needed - as that's default - so, first attempt with changing -e USERMAP_UID=0 & -e USERMAP_GID=0 only - and that worked - after turning off SELinux...

So, next todo according to sealert:

# ausearch -c 'ripmime' --raw | audit2allow -M my-ripmime
# semodule -X 300 -i my-ripmime.pp

Done! (For now.)
I'm unsure whether it would be a better to relabel the consume and export directories rather than adding an exception?:

[paperless@server ppngx]$ ls -lhZ
total 32K
-rw-r--r--. 1 paperless paperless unconfined_u:object_r:user_home_t:s0            3.3K Feb  9 20:24 backup.sh
drwxr-xr-x. 2 paperless paperless system_u:object_r:container_file_t:s0:c447,c551    6 Feb 17 19:38 consume
drwxr-xr-x. 2 paperless paperless system_u:object_r:container_file_t:s0:c447,c551    6 Feb 17 19:14 export
-rw-r--r--. 1 paperless paperless unconfined_u:object_r:user_home_t:s0            1.1K Feb  9 20:24 LICENSE
-rw-r--r--. 1 paperless paperless unconfined_u:object_r:user_home_t:s0            6.2K Feb  9 20:24 README.md
-rwxr-xr-x. 1 paperless paperless unconfined_u:object_r:user_home_t:s0            5.6K Feb 17 19:11 start.sh
-rwxr-xr-x. 1 paperless paperless unconfined_u:object_r:user_home_t:s0            4.9K Feb  9 20:24 start.sh.bak
[paperless@server ppngx]$

(Also, unsure whether my math at #17 (comment) isn't out by 1 because user 0 is the first user - root - and 999 there should be 1000 - the Paperless default Docker user of 1000? So, basically, the user mismatch reflects the difference between the default UID set for Paperless Docker (1000) and the default under Podman (0 for root for the user UID under which Rootless Podman is running...?)
Now to get rid of the piles of paper I've been collecting...
Thanks @jdoss !

0 replies

jdoss · 2024-03-08T08:14:19Z

jdoss
Mar 8, 2024
Maintainer

Wonderful! I am glad you got your issues sorted out @xpseudonym :)

0 replies

xpseudonym · 2024-03-08T15:34:05Z

xpseudonym
Mar 8, 2024
Author

@jdoss, but wait! There's more...
I just so happens, that yesterday I started scratching my head about how to implement double-sided scanning: PAPERLESS_CONSUMER_ENABLE_COLLATE_DOUBLE_SIDED=1
To implement that, the (default) directory ${PWD}/consume/double-sided needs to be created.

As a preliminary aside, I discovered some more selinux points of interest... Creating double-sided as user paperless made its selinux context unconfined_u:object_r:user_home_t:s0 whereas the selinux context of consume and export are system_u:object_r:container_file_t:s0:c447,c551. So, I went about changing its context to match consume and export. But, following directions here '4.7.2. Persistent Changes: semanage fcontext' with fcontext --equal and executing restorecon didn't change the context... It turned out that executing restorecon on directory consume actually changed its context to unconfined_u:object_r:user_home_t:s0!!! So, the system_u context of consume and export is not set permanently by semanage but temporarily by chcon... Hmm (scratches head). So, I've set double-sided to system_u with chcon - but, equally, I've no idea if that makes any more sense than running restorecon on consume and export to set their context to unconfined_u...
Ans: It does make more sense because selinux prevents paperless-webserver from reading consume with unconfined_u instead of system_u. So, why aren't consume and export set system_u permanently? (Hmm, scratches head again.)

Anyway...

While wondering what sort of filters I'd have to add to put in .forward and what in the incoming mail they should operate on, I thought to look at plus addressing - and everything became very straight forward:-

In postfix implement plus addressing with the recipient_delimiter;
Choose your plus addresses, e.g.: [email protected] and [email protected];
Replace the .forward with two .forward+s that correspond to your chosen plus address, e.g. for the above: .forward+single-sided and .forward+double-sided with corresponding contents:
"|/usr/bin/ripmime -i - --no-nameless -d /var/home/paperless/ppngx/consume"
And;
"|/usr/bin/ripmime -i - --no-nameless -d /var/home/paperless/ppngx/consume/double-sided"
Respectively.
Set you scanner to email single sided scans to [email protected] and double sided scans to [email protected];
Add environment variable options -e PAPERLESS_CONSUMER_ENABLE_COLLATE_DOUBLE_SIDED=1 and -e PAPERLESS_CONSUMER_RECURSIVE=1 to the podman create paperless command in start.sh.

It works.
Neat.

This also has the advantage that should user paperless get any standard system mail to neither of the above plus addresses, then that will be delivered in the normal way (without any complicated filtering in .forward).

Attached my commented and amended start.sh for reference:
start.sh.txt

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

getting Paperless working with locally delivered mail attachements? #17

{{title}}

Replies: 10 comments 5 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

getting Paperless working with locally delivered mail attachements? #17

xpseudonym Feb 6, 2024

Replies: 10 comments · 5 replies

jdoss Feb 6, 2024 Maintainer

xpseudonym Feb 6, 2024 Author

xpseudonym Feb 9, 2024 Author

jdoss Feb 10, 2024 Maintainer

xpseudonym Feb 13, 2024 Author

xpseudonym Feb 13, 2024 Author

jdoss Feb 13, 2024 Maintainer

xpseudonym Feb 15, 2024 Author

xpseudonym Feb 16, 2024 Author

jdoss Feb 16, 2024 Maintainer

jdoss Feb 16, 2024 Maintainer

xpseudonym Feb 17, 2024 Author

xpseudonym Feb 18, 2024 Author

jdoss Mar 8, 2024 Maintainer

xpseudonym Mar 8, 2024 Author

xpseudonym
Feb 6, 2024

Replies: 10 comments 5 replies

jdoss
Feb 6, 2024
Maintainer

xpseudonym
Feb 6, 2024
Author

xpseudonym
Feb 9, 2024
Author

jdoss
Feb 10, 2024
Maintainer

xpseudonym
Feb 13, 2024
Author

xpseudonym
Feb 13, 2024
Author

jdoss Feb 13, 2024
Maintainer

xpseudonym
Feb 15, 2024
Author

xpseudonym Feb 16, 2024
Author

jdoss Feb 16, 2024
Maintainer

jdoss Feb 16, 2024
Maintainer

xpseudonym Feb 17, 2024
Author

xpseudonym
Feb 18, 2024
Author

jdoss
Mar 8, 2024
Maintainer

xpseudonym
Mar 8, 2024
Author