Optimizing for Large Libraries?

[meowmix]

Hello. I run Poweramp with a very large library (~700GiB, 130,000 entries) of QAAC-encoded MP4 files. This is a recent change for me from around 50,000. I am curious if there are ways to improve performance on libraries of this size. With the 50k library size, my cheap phone (moto g fast) still worked okay. With the recent not-quite-tripling, UI responsiveness has dramatically changed for me. The change is much worse than a tripling. (Not an O(n) algorithm somewhere, apparently?)

The slowdown seems to be with respect to database access. When everything is loaded and in cache, Poweramp is acceptably responsive. Not instant, sure. The issue is when things get unloaded, it seems. There are times when I'll go to the Library and tap on "Recently Played" or "Folders Hierarchy" (which are my two most accessed Library entries) and it will be minutes before Poweramp enters. This isn't the only place where I encounter such slowdown. One of the most dramatic instances was when I attempted to show Info/Tags on a tune. I tapped it, set my phone down. Three songs later, the info box popped up, with the info for the song playing when the box opened. I wish there was some way to tell Android to get rekt and keep the appropriate parts of Poweramp cached in RAM or what-have-you.

I imagine that I am not the first to encounter such issues, but a Google search isn't really helping me with these issues. I'm willing to go to some extent to try help resolve this. I have a programming background (not Android, alas), so let me know what I can do to help out and make ameliorating this issue as easy as possible. I've spent half a year poking around now, trying to help myself, and I'm not succeeding!

I always find the failure profile of software to be instructive as to its quality, and the funny part about all of this is that Poweramp does a great job of continuing to play music despite whatever this issue is. Good job in this regard!

[andrewilley]

Android's recommended Storage Access Framework is not exactly the fastest storage processing solution in the world - far from it in fact! So you could try using Legacy File Access Mode instead, which is now located in PA Settings=>Library, but used to be in PA Settings=>Misc on older versions. This will require you to reassign your choice of Music Folders again, and be aware that this will trigger a new full rescan which might take a while. Make a backup of any needed playlists, ratings, etc before you start.

Also once scanned, you might want to turn off the Auto Scan option (in PA Settings=>Library=>Scanner) in case it is getting clogged up with rescans.

But otherwise, the issues you describe are not normal but as you say your phone is fairly basic in terms of its available working RAM for example. But three songs later to show Info/Tags is seriously messed up, as that one should be pretty much instant as it only needs to read the headers read from the current file.

Andre

[meowmix]

Sorry for the long response time. Been a busy summer and I haven't been listening to music as often.

https://www.youtube.com/watch?v=shNJklvj9bA - I've recorded a screengrab of the behaviour I've described. I switched to Legacy File Access, which resulted in a slower rescan, but whatev. I had Auto Scan already disabled. Thank you for the suggestions. They did not improve behaviour substantially. Seems to be a bit of a better behaviour in the worst case scenario.

Here's the list of timestamps from the video.

0:07 - app open
0:11 - first metadata load
0:24 - Recently Played attempted access
2:23 - bail out
2:28 - back to main player UI
3:08 - metadata loads again, try to load metadata browser
3:50 - attempt Info/Tags again
6:06 - Info/Tags loads
6:15 - Recently Played attempt 2
9:06 - Recently Played loads

Notably, there's a 2 minute delay for Info/Tags opening, and just shy of a 3 minute delay for Recently Played. Seems weird if there's a 2 minute delay to open a single file's metadata.

My next attempt at resolution will be returning to Opus to see if the change in metadata improves anything. I will report back after the attempt.

Thanks again for the help.

[andrewilley]

Definitely not what you would expect, even with a large library where the scan times might be longer (but you've turned AutoScan off, so that should not be an issue after the first one has been completed).

Might be worth sending a logfile immediately after an Info/Tags display eventually shows up - use PA Settings=>Get Support=>Send Log to gpmaxmpz [at] gmail.com, and let Max know you've sent it by posting a message here or DMing @maxmp.

Andre

[maxmp]

I would add that for the initial scan (right after Poweramp is installed, or if Full Rescan is initiated), try to keep Poweramp playing something/on the background, as scanning large library on slower storage (sd-card) may take some time and Android will kill Poweramp if it's not playing something. Poweramp will attempt to scan further on the next run still, but you may not get all your library available until it finishes the scan.
There are plans to improve this - by showing "Scan in progress" notification.

[Luis David García Solano]

@maxmp yes please I also have a very big library about 70000 files, even I have devided into folders of a 1000 songs each one, the full scan takes kind of 4 hours and put the scanning in a background progress will help a lot while using the phone in other activities.

[andrewilley]

Even 1000 files per folder is pretty big. My only folder that's anywhere near that big is my Singles folder, with everything else well under 100 per folder. Any larger and I'd create themed subfolders to make browsing practical.

Andre

[MotleyG]

1 hour ago, Luis David García Solano said:

@maxmp yes please I also have a very big library about 70000 files, even I have devided into folders of a 1000 songs each one, the full scan takes kind of 4 hours and put the scanning in a background progress will help a lot while using the phone in other activities.

This still Seems pretty slow. Are there any other reasons it could be taking this long? Things like missing tags or covers may add some time, but generally this is after the initial scan.

I have 25k tracks with a mix of ALAC and mp3. It typically takes about 12-15 minutes for a Full Rescan. So 70k should take somewhere between 36-45 minutes I would think. 

[Luis David García Solano]

@andrewilley do you think that dividing the folders with 100 tracks each will make it faster?

1 hour ago, andrewilley said:

Even 1000 files per folder is pretty big. My only folder that's anywhere near that big is my Singles folder, with everything else well under 100 per folder. Any larger and I'd create themed subfolders to make browsing practical.

Andre

 

[andrewilley]

@Luis David García Solano possibly, although I agree with @MotleyG that 4 hours seems very long even for that number of items. I have about 7,000 songs and it takes just a few minutes to scan, so half to three-quarters of an hour would seem more expected. But I use folders and subfolders organised by artist, album, etc.

What storage media are you using, and what access type (Legacy File Access Mode, or modern Android?)

Andre

[MotleyG]

1 hour ago, Luis David García Solano said:

@andrewilley do you think that dividing the folders with 100 tracks each will make it faster?

It might help, but again it also depends on how complete your metadata tags are, and what PA may have to guess at. There are some good tag editors that will make it easy to reorganize them into Albumartist and Album to really simplify things.

BTW - With that many files I would highly recommend that you DISABLE the autoscan options for the library. If You make any changes, you can manually start a refresh. But if you leave it on auto, PA will be rescanning through all those files on every startup and slowing things down.

[meowmix]

After attempting to reproduce this for a proper bug report, I've been unable to do so. Performance is back to comparable to 50k. I believe that max was correct in his presumption that it had to do with scanning.

Thank you all for your patience and help!

Now to transcode my module collection and add another ~350k songs to the mix! Ahahaha! ❤️