Allow using Downloads as a storage location

[AlvaroBrey]

On Android >= 10, MediaScanner won't index (either manually or automatically) files in our app's private folder. That means that other apps can't access files downloaded in Nextcloud.

To solve this, we intend to allow setting Downloads as a storage directory, which would be public.

• Add option to set storage directory to Downloads.
  • Requires WRITE_EXTERNAL_STORAGE or MANAGE_EXTERNAL_STORAGE as we use the File API. Could be done without it using the MediaStore API directly but this is an unknown amount of work.
  • Test what happens if files are removed externally
  • Should we make this the default?
  • Test with android 9, 10, 11 & 12
  • Add all files to MediaScanner if in Downloads
    - [ ] Media files may need to be added twice? (once to the Downloads collection and another to media)
    - [ ] Ensure files are deleted from MediaScanner when deleted

Fixes #6150
Fixes #10038

• Tests written, or not not needed

[github-actions]

APK file: https://www.kaminsky.me/nc-dev/android-artifacts/9981.apk



To test this change/fix you can simply download above APK file and install and test it in parallel to your existing Nextcloud app.

[nextcloud-android-bot]

Codacy

Lint

Type	master	PR
Warnings	92	92
Errors	0	0

SpotBugs (new)

Warning Type	Number
Bad practice Warnings	28
Correctness Warnings	78
Experimental Warnings	1
Internationalization Warnings	9
Malicious code vulnerability Warnings	57
Multithreaded correctness Warnings	9
Performance Warnings	67
Security Warnings	29
Dodgy code Warnings	350
Total	628

SpotBugs (master)

Warning Type	Number
Bad practice Warnings	28
Correctness Warnings	78
Experimental Warnings	1
Internationalization Warnings	9
Malicious code vulnerability Warnings	57
Multithreaded correctness Warnings	9
Performance Warnings	67
Security Warnings	29
Dodgy code Warnings	350
Total	628

[rmelotte]

I tested the APK linked above on an Android 11 device (LineageOS - Fairphone 3), I'll detail below what I observed on my device.

At first the Downloads folder option didn't show up in the list, even though I gave the app "Allow management of all files".
I fiddled with the permissions a bit (switching from "Allow management of all files" to "Allow access to media only" and back).
Then I forced closed the app and opened it again, and the Downloads folder option appeared.

I configured the Screenshots folder to be auto-uploaded, with files moved to the App folder (now Downloads).
New screenshots do get uploaded and moved in the correct place (/storage/emulated/0/Download/nextcloud/user@domain/InstantUpload/Screenshots/Screenshot_20220330-145428_Nextcloud_QA.png).

However, in the default gallery app that comes with LineageOS, they show up as all-grey images, and the reported path is:

/storage/emulated/0/(invalid)/nextcloud/user@domain/insteadUpload/Screenshots/Screenshot_20220330-145428_NextcloudQA.png/Screenshot_20220330-145428_Nextcloud_QA.png

Note that the additional /Screenshot_20220330-145428_NextcloudQA.png/ is not a typo: the gallery app reports an additional folder whose name is the same as the image name.

[AlvaroBrey]

I tested the APK linked above on an Android 11 device (LineageOS - Fairphone 3), I'll detail below what I observed on my device.

At first the Downloads folder option didn't show up in the list, even though I gave the app "Allow management of all files". I fiddled with the permissions a bit (switching from "Allow management of all files" to "Allow access to media only" and back). Then I forced closed the app and opened it again, and the Downloads folder option appeared.

I configured the Screenshots folder to be auto-uploaded, with files moved to the App folder (now Downloads). New screenshots do get uploaded and moved in the correct place (/storage/emulated/0/Download/nextcloud/user@domain/InstantUpload/Screenshots/Screenshot_20220330-145428_Nextcloud_QA.png).

However, in the default gallery app that comes with LineageOS, they show up as all-grey images, and the reported path is:

/storage/emulated/0/(invalid)/nextcloud/user@domain/insteadUpload/Screenshots/Screenshot_20220330-145428_NextcloudQA.png/Screenshot_20220330-145428_Nextcloud_QA.png

Note that the additional /Screenshot_20220330-145428_NextcloudQA.png/ is not a typo: the gallery app reports an additional folder whose name is the same as the image name.

Thanks for the analysis @rmelotte, this is still a work in progress but that info will help us get it done

[AlvaroBrey]

@tobiasKaminsky perhaps we should split this in two:

1. Disable mediaScanner on android >= 10. This will not fix media scanner (as files will still be in private folder) but at least we won't get the (invalid) folder bug. This should be fairly easy (couple of hours work, maybe) and could be done ASAP.
2. Allow storing files in Downloads or SAF chosen folder. This would solve media scanner potentially but would take much more work. We could leave this for later.

[github-actions]

Codacy

Lint

Type	master	PR
Warnings	85	85
Errors	0	0

SpotBugs

Category	Base	New
Bad practice	33	33
Correctness	45	45
Dodgy code	351	351
Experimental	1	1
Internationalization	9	9
Multithreaded correctness	9	9
Performance	65	65
Security	28	28
Total	541	541

[AlvaroBrey]

perhaps we should split this in two:

1. Disable mediaScanner on android >= 10. This will not fix media scanner (as files will still be in private folder) but at least we won't get the (invalid) folder bug. This should be fairly easy (couple of hours work, maybe) and could be done ASAP.

2. Allow storing files in Downloads or SAF chosen folder. This would solve media scanner potentially but would take much more work. We could leave this for later.

1. Is done via #10353 ; changed PR title and description to reflect this. This PR will now be just for adding Downloads as a storage location.

[codecov]

Codecov Report

Merging #9981 (da591f2) into master (82d86b9) will increase coverage by 0.01%.
The diff coverage is 15.38%.

@@             Coverage Diff              @@
##             master    #9981      +/-   ##
============================================
+ Coverage     31.25%   31.26%   +0.01%     
+ Complexity     3228     3227       -1     
============================================
  Files           536      536              
  Lines         39667    39671       +4     
  Branches       5462     5462              
============================================
+ Hits          12396    12404       +8     
+ Misses        25459    25455       -4     
  Partials       1812     1812              

Impacted Files	Coverage Δ
...cloud/android/datastorage/DataStorageProvider.java	62.50% <10.00%> (-9.60%)	⬇️
.../java/com/owncloud/android/utils/PermissionUtil.kt	9.45% <33.33%> (+0.12%)	⬆️
...main/java/com/owncloud/android/utils/FileUtil.java	60.00% <0.00%> (-26.67%)	⬇️
...va/com/owncloud/android/utils/FilesSyncHelper.java	14.60% <0.00%> (-6.75%)	⬇️
...cloud/android/datamodel/UploadsStorageManager.java	66.42% <0.00%> (-5.78%)	⬇️
...om/owncloud/android/utils/ErrorMessageAdapter.java	3.36% <0.00%> (-3.85%)	⬇️
.../owncloud/android/files/services/FileUploader.java	48.48% <0.00%> (-3.24%)	⬇️
...oud/android/operations/RefreshFolderOperation.java	76.15% <0.00%> (-1.43%)	⬇️
...ain/java/com/owncloud/android/db/UploadResult.java	39.74% <0.00%> (-1.29%)	⬇️
...a/com/owncloud/android/utils/FileStorageUtils.java	52.29% <0.00%> (-1.15%)	⬇️
... and 12 more

[damien-sorin]

Hi
and thanks at first for targeting the MediaScanner Issue which is really annoying!

I am just wondering why the solutions seems to be "choosing an other storage folder" - yeah I understand that "Downloads" has some kind of special handling by the system, but there are two things at least at my two devices:

• After downloading files with nextcloud they (of course) don't appear - ok, nothing new, but after a rename of the file itself or one of its parent folders (and renaming them back after that) the file will appear immediately in gallery and MediaStorage related apps

The rename could be done with a file explorer or with "mv" via adb shell.

• Other sync apps like FolderSync and AutoSync let me configure any custom folder I wish, such as "/sdcard/MyMediaFiles" and everything synced there will appear immediately in gallery and MediaStorage related apps

So from my point of view there has to be a solution for the original storage folders of nextcloud, too. At least for

• /sdcard/nextcloud/
• /sdcard/Android/media/com.nextcloud.client/...

Sorry if I am telling nothing new to you, but probably there is some new information helping you to get a full solution for the problem.

[simonspa]

I can confirm this behavior - even more, if I delete the created (invalid) folder and load the thumbnails of images via the Nextcloud Android Media tab, they also start showing up in the gallery app, with the correct path (.../Android/media/com.nextcloud.client/nextlcoud/...).

[simonspa]

...and a few more observations:

• As long as there is an (invalid) folder present on the system, the Media tab of Nextcloud would not show any thumbnails.
• After deleting the folder, thumbnails were shown - but for new photos, they are gone again while there is no (invalid) folder to delete

[damien-sorin]

ah, and please make 'Downloads' not the default, because i really like it to have

/sdcard/nextcloud/

as storage folder, which is already difficult to configure (you have uninstall the app completely and reinstall, after that never touch the storage config. then nextcloud will use this folder because none of the three options is chosen).

or add that folder to the available storage folders, too, because its already possible to use.

[AlvaroBrey]

Sorry if I am telling nothing new to you, but probably there is some new information helping you to get a full solution for the problem.

Thanks for the info; this is a complicated issue to work on due to differences between android versions and manufacturers. Some more research is needed; it looks like in some phones, files in Android/media (but not Android/data) can be scanned.

[damien-sorin]

Sorry if I am telling nothing new to you, but probably there is some new information helping you to get a full solution for the problem.

Thanks for the info; this is a complicated issue to work on due to differences between android versions and manufacturers. Some more research is needed; it looks like in some phones, files in Android/media (but not Android/data) can be scanned.

yeah, i can imagine. its like website dev in 2002 😅 probably worse.

for me it was important to give the information about the "rename"-hack because that works at least on huawei p20 and oneplus 9 with latest android uodates installed.

if i can solve it with some manual hack probably the app can do it automatically (probably with more elegance 😉)

[AlvaroBrey]

if i can solve it with some manual hack probably the app can do it automatically (probably with more elegance wink)

Quick question, does a reboot of the phone also do the trick, like renaming?

[damien-sorin]

if i can solve it with some manual hack probably the app can do it automatically (probably with more elegance wink)

Quick question, does a reboot of the phone also do the trick, like renaming?

Quick answer: No.

the media scanner seems not be triggered on reboot anymore.

only triggering filesystem events seems to work.

i had a talk to the the author of AutoSync, and he says the app downloads the file as a tmp file, then deletes the original and then renames the tmp file to the original file.

[simonspa]

@AlvaroBrey on my Pixel 4, Android 12, a reboot adds the pictures to the gallery, so apparently the media scanner discovers them.

[nkming2]

I think moving to /Download is the way to go now with Google enforcing scoped storage. Even if some manufacturers today are not enforcing it correctly, it's probably a bug and Google will make them fix it sooner or later.

As a side note, Google drive is also downloading to /Download

[damien-sorin]

from an current point of view, the nextcloud client is the only one app with this problem besides many synchronization apps which all download the files anywhere else and get there files indexed.

sorry, but download folder is only the path of least resistance to say "it works, if you dont like it use another app"

[nkming2]

from an current point of view, the nextcloud client is the only one app with this problem besides many synchronization apps which all download the files anywhere else and get there files indexed.

sorry, but download folder is only the path of least resistance to say "it works, if you dont like it use another app"

That's interesting, wonder how the other clients work, because from what I understand from the Android update note here, it doesn't seems to be allowed.

[damien-sorin]

i don't know how they do this, but it works really good on my android 12 device.

i just swiched to AutoSync because it offers functionality like the nextcloud client for windows.

With it (for example) i just sync my local DCIM folder, including instant upload if i make a photo and sync down from remote folder if i have made photos on a different device.
everything is indexed immediately and visible to any otther app.

the nextcloud app is now used for browsing the nextcloud for single files only. in this use case the download folder would be okay a local folder.
but in a real sync use case it isnt.

[nkming2]

What you want is two-way sync which is a slightly different issue and should be handled differently. Other apps can see the images inside DCIM because DCIM is a standard directory that the MediaStore API supports for images and videos. But I think this issue is more about downloading an arbitrary file and making it available to other apps.

[damien-sorin]

sorry, but you dont get my point if you only read my last post.

what I mean is that is is not relevant where the other apps store the file, images are immediately indexed and visible within gallery apps and so on.

it works with every non standard path.

just an example from my phone

/storage/emulated/0/SyncTest_Nextcloud/

[rhatguy]

Rather than trying to handle every possible use case up front would it be better to enable the Downloads folder for now? The existing setup with the private app folders effectively puts Nextcloud in a "jail" on any non-technical users phone completely destroying the WAF of Nextcloud.

[flan7]

Rather than trying to handle every possible use case up front would it be better to enable the Downloads folder for now? The existing setup with the private app folders effectively puts Nextcloud in a "jail" on any non-technical users phone completely destroying the WAF of Nextcloud.

+1 to fixing the 2 year old regression bug instead of debating how for 6 months. Its already broken beyond use, so why not make it usable.