Repository navigation
[Bug]: enabling external storage on 28.0.2 generates error during occ files:scan #43408
Description
⚠️ This issue respects the following points: ⚠️
- This is a bug, not a question or a configuration/webserver/proxy issue.
- This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
- Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
- I agree to follow Nextcloud's Code of Conduct.
Bug description
New / fresh installation of Nextcloud 28.0.2
Steps to reproduce
- Fresh install of 28.0.2
- Enable external storage
- Configure S3 object storage
- Add user with access to S3 storage
- Run nocc-cldtst01 files:scan timv (alias to below)
alias nocc-cldtst01="sudo -u www-data php /var/www/cldtst01.site1.com/html/occ"
Expected behavior
Output
root@X.X.X.X:$ nocc-cldtst01-site1 files:scan --all$
Starting scan for user 1 out of 1 (admin)
Error during scan: Array to string conversion
Error during scan: Array to string conversion
....................
Error during scan: Array to string conversion
Error during scan: Array to string conversion
+---------+-------+-----+---------+---------+--------+--------------+
| Folders | Files | New | Updated | Removed | Errors | Elapsed time |
+---------+-------+-----+---------+---------+--------+--------------+
| 49 | 218 | 0 | 1 | 0 | 84 | 00:00:01 |
+---------+-------+-----+---------+---------+--------+--------------+
root@X.X.X.X:
Installation method
Community Manual installation with Archive
Nextcloud Server version
28
Operating system
Debian/Ubuntu
PHP engine version
PHP 8.2
Web server
Nginx
Database engine version
MariaDB
Is this bug present after an update or on a fresh install?
Fresh Nextcloud Server install
Are you using the Nextcloud Server Encryption module?
Encryption is Disabled
What user-backends are you using?
- Default user-backend (database)
- LDAP/ Active Directory
- SSO - SAML
- Other
Configuration report
root@X.X.X.X:~$ nocc-cldtst01-site1 config:list system
{
"system": {
"passwordsalt": "***REMOVED SENSITIVE VALUE***",
"secret": "***REMOVED SENSITIVE VALUE***",
"trusted_domains": [
"vmi1517.site1.com",
"cldtst01.site1.com",
"X.X.X.X"
],
"datadirectory": "***REMOVED SENSITIVE VALUE***",
"dbtype": "mysql",
"version": "28.0.2.5",
"overwrite.cli.url": "https:\/\/cldtst01.site1.com",
"dbname": "***REMOVED SENSITIVE VALUE***",
"dbhost": "***REMOVED SENSITIVE VALUE***",
"dbport": "",
"dbtableprefix": "oc_",
"mysql.utf8mb4": true,
"dbuser": "***REMOVED SENSITIVE VALUE***",
"dbpassword": "***REMOVED SENSITIVE VALUE***",
"installed": true,
"instanceid": "***REMOVED SENSITIVE VALUE***",
"overwritehost": "cldtst01.site1.com",
"activity_expire_days": 14,
"allow_local_remote_servers": true,
"auth.bruteforce.protection.enabled": true,
"blacklisted_files": [
".htaccess",
"Thumbs.db",
"thumbs.db"
],
"cron_log": true,
"default_phone_region": "US",
"enable_previews": true,
"enabledPreviewProviders": [
"OC\\Preview\\PNG",
"OC\\Preview\\JPEG",
"OC\\Preview\\GIF",
"OC\\Preview\\BMP",
"OC\\Preview\\XBitmap",
"OC\\Preview\\Movie",
"OC\\Preview\\PDF",
"OC\\Preview\\MP3",
"OC\\Preview\\TXT",
"OC\\Preview\\MarkDown"
],
"filesystem_check_changes": 0,
"filelocking.enabled": "true",
"htaccess.RewriteBase": "\/",
"integrity.check.disabled": false,
"knowledgebaseenabled": false,
"log_rotate_size": "104857600",
"logfile": "\/var\/log\/nextcloud\/cldtst01-site1.log",
"loglevel": 2,
"logtimezone": "America\/Los_Angeles",
"memcache.local": "\\OC\\Memcache\\APCu",
"memcache.locking": "\\OC\\Memcache\\Redis",
"overwriteprotocol": "https",
"preview_max_x": 1024,
"preview_max_y": 768,
"preview_max_scale_factor": 1,
"profile.enabled": false,
"redis": {
"host": "***REMOVED SENSITIVE VALUE***",
"port": 0,
"password": "***REMOVED SENSITIVE VALUE***",
"timeout": 0.5,
"dbindex": 1
},
"quota_include_external_storage": false,
"share_folder": "\/StorageShare",
"skeletondirectory": "",
"trashbin_retention_obligation": "auto, 7"
}
}
root@X.X.X.X:~$List of activated Apps
root@X.X.X.X:~$ nocc-cldtst01-site1 app:list
Enabled:
- activity: 2.20.0
- admin_audit: 1.18.0
- cloud_federation_api: 1.11.0
- dav: 1.29.1
- federatedfilesharing: 1.18.0
- files: 2.0.0
- files_external: 1.20.0
- files_pdfviewer: 2.9.0
- files_sharing: 1.20.0
- files_trashbin: 1.18.0
- files_versions: 1.21.0
- logreader: 2.13.0
- lookup_server_connector: 1.16.0
- oauth2: 1.16.3
- password_policy: 1.18.0
- photos: 2.4.0
- privacy: 1.12.0
- provisioning_api: 1.18.0
- serverinfo: 1.18.0
- settings: 1.10.1
- sharebymail: 1.18.0
- text: 3.9.1
- theming: 2.3.0
- twofactor_backupcodes: 1.17.0
- updatenotification: 1.18.0
- viewer: 2.2.0
- workflowengine: 2.10.0
Disabled:
- bruteforcesettings: 2.8.0
- circles: 28.0.0-dev (installed 28.0.0-dev)
- comments: 1.18.0 (installed 1.18.0)
- contactsinteraction: 1.9.0 (installed 1.9.0)
- dashboard: 7.8.0 (installed 7.8.0)
- encryption: 2.16.0
- federation: 1.18.0 (installed 1.18.0)
- files_reminders: 1.1.0 (installed 1.1.0)
- firstrunwizard: 2.17.0 (installed 2.17.0)
- nextcloud_announcements: 1.17.0 (installed 1.17.0)
- notifications: 2.16.0 (installed 2.16.0)
- recommendations: 2.0.0 (installed 2.0.0)
- related_resources: 1.3.0 (installed 1.3.0)
- support: 1.11.0 (installed 1.11.0)
- survey_client: 1.16.0 (installed 1.16.0)
- suspicious_login: 6.0.0
- systemtags: 1.18.0 (installed 1.18.0)
- twofactor_totp: 10.0.0-beta.2
- user_ldap: 1.19.0
- user_status: 1.8.1 (installed 1.8.1)
- weather_status: 1.8.0 (installed 1.8.0)
root@X.X.X.X:~$Nextcloud Signing status
No errors have been found.Nextcloud Logs
root@X.X.X.X:/var/log/nextcloud$ tail -f cldtst01-site1.log
{"reqId":"BsKXnLgxxgk9OrTCg4dN","level":2,"time":"2024-02-06T13:25:37-08:00","remoteAddr":"Z.Z.Z.Z","user":"--","app":"no app in context","method":"POST","url":"/login","message":"Login failed: timv (Remote IP: Z.Z.Z.Z3)","userAgent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36","version":"28.0.2.5","data":[]}Additional info
No response
Activity
joshtrichards commentedon Feb 7, 2024
Unable to reproduce. Tried both without and with files in the bucket. Can you please:
-vvvto see what files/folders it is hitting these errorsocc files_external:list --output=json_pretty(remove any secrets/info you deem confidential)For the record, what S3 back-end (or provider) are you using?
[-][Bug]: enabling external storage on 28.0.2 generates error[/-][+][Bug]: enabling external storage on 28.0.2 generates error during `occ files:scan`[/+]nfguide commentedon Feb 7, 2024
Here is a snippet of the scan:
Thanks for taking the time to review. There is another config with 27.1.6 connecting to the same object storage, and the scan works correctly, with no errors.
mhzawadi commentedon Feb 7, 2024
Can confirm this is a bug with S3 like buckets, I have the same issue. I am using Backblaze B2 as the bucket
amcguire10 commentedon Feb 8, 2024
I am also confirming this issue with S3. Using Dreamhost's DreamObjects. "Identical" error list as above except for the line(s) with "Scanner.php(400)". Those are not included. This occurred after updating Nextcloud from 25.0.2.3 to 28.0.2.5 (mutiple upgrade steps) and php from 8.0 to 8.2.
The log seems to indicate that the issue started after the core upgrade from 27.1.6.2 to 28.0.2.5.
nfguide commentedon Feb 8, 2024
I concur the started "after the core upgrade from 27.1.6.2 to 28.0.2.5", as the error is not present on a test config running 27.1.6.
Benance commentedon Feb 9, 2024
hello
I can reproduce this error when I launch occ files:scan with an smb external storage too
stratacorps commentedon Feb 9, 2024
Not sure if it helps to add another voice, but am tracking this. Exact same issue as OP so not posting any new details unless it helps.
27 remaining items
ahcm commentedon Apr 3, 2024
If you think running occ:files scan making user's data disappear to them is not highest priority, you have never run a service with a large user base.
stratacorps commentedon Apr 3, 2024
What data loss or data disappearing is occurring with this bug?
ahcm commentedon Apr 3, 2024
The users that are affected will not see any files.
Removing the external storage and using occ:files scan again will make them see their files again.
Now external storage is not available to the users.
joshtrichards commentedon Apr 3, 2024
And with #43794 in place? The fix has been available for over a month. Others have stated it addresses this matter. I just haven't had time to adjust it (so it doesn't rely on serialization), which is an optimization preferred before merging formally.
Commercial clients of Nextcloud GmbH already have a dedicated channel for bug reporting, escalation, etc. It's outside of GitHub via https://portal.nextcloud.com.
For the record, I'm a member of the open source Nextcloud project, but I'm not employed by Nextcloud GmbH.
lavigne958 commentedon Apr 3, 2024
and thank you for taking the time to fix this issue. I tried it myself and could not figure it out (I'm too new to Nextcloud development)
The fix works in term of error rate, I have some external storage connected to my nextcloud but I'm too afraid to remove it and potentially lose it all 😬 or need to upload everything again 😞 , too risky for me to test it.
If that works I can connect a new external storage and check the behavior (run a scan, remove it, run a scan, connect it again, run a scan etc), if this has no effect on other external storages already connected 🙄
nfguide commentedon Apr 4, 2024
@lavigne958
The sledgehammer approach I used was to rollback to 27.x and will stay on 27 until 30.x is released which properly addresses the issue. I did taker a look at the patch mentioned above, and as I am NOT a coder and NOT completely knowledgeable enough to sort through the .php file to slap some code into file on the hopes it works, 27.x was the best option available.
Additionally, it's not real clear what the advantages are moving from 27.x when the primary use is online file storage.
Looking forward to 30.x.
compuguy commentedon Apr 30, 2024
Now I cant speak for larger deployments, but I was able to apply the patch to the latest 28.x release. I've not seen any issues on my instance (knock on wood).....
HomelabHaven commentedon May 3, 2024
Was hoping the issue would be solved in version 29.0.0 as I read in other related threads... guess not:
Hard to troubleshoot when the system log is being inundated with "Scanner.php" S3 related error messages.
I NEVER change anything with my nextcloud setup except for version updates which seems to break something different every time. Quite disappointing when the system as a whole has so much potential yet I had the same buggy experience 7 years ago when trying nextcloud for the first time and giving it up shortly thereafter.
fix(Files): Change how scanner diffs for changed metadata
fix(Files): Change how scanner diffs for changed metadata
fix(Files): Change how scanner diffs for changed metadata
oadaeh commentedon May 8, 2024
Thanks for the patch. I've been dealing with random sync errors for a few months, and after applying the patch to address the log errors, I think the sync errors have gone away, too. I only just applied the patch about an hour ago, so there still might be something else, but so far, it's looking pretty good.
fix(Files): Change how scanner diffs for changed metadata
fix(Files): Change how scanner diffs for changed metadata