Collection update best practices

[Arkats]

Hello guys,

I have a large music collection (36K tracks) and all tracks are carefully tagged. I manage everything with MusicBee on my PC. Then I use a synchronization app to update my MicroSD card from my music folder.

After some time using PA, I have decided that the best way was to re-scan entirely my collection each time I was doing an update (adding new albums) because just doing a simple scan was not perfect, some tracks were missing (as far I remember).

But today, I have a problem with sorting my album by date of added to the collection as some albums are presented recent when they are not and I guess that is may be because each time I do a total re-scan (even is the sorting is not completely wrong...)

So, what would be the best practice to keep my collection properly updated on PA?

Thank you for your help!

[6b6561]

Hi,

A long time ago I had some issues with PA indexing files before the metadata blocks of the files had completely been copied to the device, and this lead to metadata not showing up in PA. The easy fix was to do a full re-scan, but that causes issues with dates added. The other fix was to rename the folder which causes PA to re-scan just the files in the folder.

You don't mention how you are doing the sync, but if you connect your phone with cable and synchronizes the directory from PC to phone, then it might be so slow that you run into this situation.

What I would then recommend  you to do is to disable the automatic scans in PA, copy the files to the device and once the copy is done, then just run a normal scan. This should work.

Do you have all 36k files in the same folder? As such modern file systems should be able to coupe with it, but I would still recommend use of sub folders.

[6b6561]

I have ~15k songs and I have my music library folder shared from my PC and use FolderSync Pro on my devices to handle the synchronization. I have used this for years on numerous devices without any issues. (There is a setting in FolderSync that makes it use a temporary file, this ensures that the file is fully copied before PA scans the meta data)

[Arkats]

Hello 6b6561,

Thank you for your reply.

To synchronize my PC Music folder on my MicroSD, I plug the card directly on my PC because the synchronization app I use (Synchredible) is not able to do the job if the MicroSD card is in the phone. Not necessarily practical but at the end I am fine with that knowing that after having tried many different apps that is the one I prefer. But I didn't know about FolderSync Pro and it seems to be able to do it, so I will give a try... but as you mentioned it, this might make the synchro much more slow... So, to be tested.

Yes, my Music folder on PC contains sub folders by artists. It is managed directly by MusicBee.

As you mentioned the automatic scan in PA, I have tried it at the beginning and I was not satisfied because some files were missing, so It is off.

At the end, next time I will try again just a simple scan instead of a total re-scan, and I will see if it is fine.

Of course, as my microSD is out of my phone for synchro, I do not have the problem you mentioned with PA scanning when the synchro is not completed. Anyhow, as the auto scan is off, it is a manual scan, which gives you the hand on when to do the scan.

Thank you very much for your help!!

 

[andrewilley]

First, I generally don't advise using automated sync software. It may work fine, but many users experience issues with the process being a bit too automated and doing things out of their control. Apps like MusicBee seem to have a mind of the own sometimes and can be difficult to tame, and we can't support whatever they get up to. I tend to just copy files over the old-fashioned manual way.

That said, I do personally use FolderSync (with "Use temp-file scheme" enabled) for some things (such as regularly downloaded podcasts) which seems to do a good job as it just accurately copies any changed files (audio or otherwise) without affecting their contents or their last-modified timestamp.

The "Date First Added to Library" information stored in PA's database should not be wiped by a Full Rescan as it remembers the names and locations of any files that it had previously found. Thus you can modify the contents of a file (audio, metadata tags, etc) just as you wish and keep the same "First Added" date. Minor modifications, such as editing only a single tag, are sometimes best picked up by doing a Full Rescan.

HOWEVER if you change anything about the filename or its storage path, PA will assume the old file has been removed/deleted, and it will see a new file to be scanned into its database - thus it will have a new First Added date. So "Date Added" will not be retained if you change a filename or the folder path that it is contained within.

Andre

[Arkats]

18 hours ago, andrewilley said:

It may work fine, but many users experience issues with the process being a bit too automated and doing things out of their control.

I guess it is true as over the years I have used and test many different synch apps and at one moment or another I started to have problems. Things have changed when I have used "Synchredible" that is so far very robust and trustworthy. I will check on FolderSync though.

 

18 hours ago, andrewilley said:

MusicBee seem to have a mind of the own sometimes

Out of curiosity, what music database software do you use?

 

18 hours ago, andrewilley said:

The "Date First Added to Library" information stored in PA's database should not be wiped by a Full Rescan

That is a good news, and I was suspecting it as the sorting was quite accurate but with some mistakes that I cannot really understand why.

 

18 hours ago, andrewilley said:

Thus you can modify the contents of a file (audio, metadata tags, etc) just as you wish

How can I do that to modify the "Date First Added to Library"? Because the best I have found it is to go on Info/tag of the actual playing song but there no access to "Date First Added to Library", only the main tags.

 

Thank you very much Andre for your help!!

[andrewilley]

4 hours ago, Arkats said:

Out of curiosity, what music database software do you use?

I don't. My audio files on my PC are organised by folder, and I navigate to them using regular file windows. 

 

4 hours ago, Arkats said:

How can I do that to modify the "Date First Added to Library"? Because the best I have found it is to go on Info/tag of the actual playing song but there no access to "Date First Added to Library", only the main tags.

"Date First Added to Library" is an internal database item, it's not intended to be user-editable. Although you might be able to do something via the API if you are a programmer, but I really won't recommend it. You can change the filesystem's last-modified timestamp with the right software though - such as using the "touch" command from a command line tool - and it's possible to sort by that if you want (a Full Rescan would probably be needed after making changes to the last-modified date). Messy though, would be easier to do it on a computer if you want to change a lot of files.

Andre

[6b6561]

@Arkats I'm also manually organizing my music, I have a Artist/Album folder setup and a tree for Compilation albums. I'm quite pedant about my tags and fix the tags when the tracks are added, if I happen to note an error, then I will screenshot it as a reminder and and some point fix the issue on the PC and then I will get the fixed tracks to the phone as part of the next sync.

@andrewilley it must be quite new that the first added date is retained through a full re-scan as in the past the albums was added in an alphabetical order after a full re-scan. I'm happy to see that the date added is retained nowadays. This has been a quite big annoyance earlier on. 

[andrewilley]

@6b6561 I think the logic was to preserve certain database-stored values which users might not want to lose - Ratings and Play Counts mainly, but I think Date Added to Library is also restored after a Full Rescan too. To completely clear them all, I would suggest setting Music Folders to a null location (so the library shows zero files, zero folders) then set it back to the real music location and scan again. 

Andre

[Arkats]

@andrewilley & @6b6561

3 hours ago, 6b6561 said:

I'm also manually organizing my music

Oh! Suddenly it seems quite austere to me. I appreciate very much MusicBee as you have nice and easy access to your collection, to tag, to rate, to generate playlists and more. But I guess you do the same with the different mp3 software you have mentioned. I am also using kid3 to go on the different levels of ID3.

@andrewilley

On 7/1/2024 at 6:40 PM, andrewilley said:

"Date First Added to Library" is an internal database item, it's not intended to be user-editable.

Too bad. And the two options you mentioned seem quite technical and I do not want to mess up.

But to be sure to understand well the "Date First Added to Library" concept, which library are we talking about? The one on my PC, so the date is the one when I have added the track on my PC music library (which seems to be the case if you mentioned "filesystem's last-modified timestamp" or the one on my phone and it is the date that PA add when it scans the collection and find new tracks?

 

[Fitzian]

@Arkats

>I am also using kid3 to go on the different levels of ID3.

Ah! You are on a mac! All our pushing to use mp3tag (and it wasn't available for your OS)

https://drive.google.com/file/d/191sB-6oAUGfg-77Azp54M8qr4W-nuB2w/view?usp=drivesdk

^ here is a link to a complete guide for kid3 if you need it.

 

[andrewilley]

1 hour ago, Arkats said:

But to be sure to understand well the "Date First Added to Library" concept, which library are we talking about? The one on my PC, so the date is the one when I have added the track on my PC music library (which seems to be the case if you mentioned "filesystem's last-modified timestamp" or the one on my phone and it is the date that PA add when it scans the collection and find new tracks?

PA doesn't know (or care) about your PC, Mac, or any other type of music collection you might have elsewhere. The "Date Added to Library" concept is simply the very first time a given file (recognised by its exact filename and precise path, not the contents) was found and scanned into PA's database. Modifying the contents of that file later won't alter the saved "first seen" date, however moving the file or renaming it will cause PA to treat it as a completely new file (with assumption that the old file has been deleted by the user).

The filesystem's "last modified" timestamp is a different thing, and that will change when you edit a file. That attribute will most likely match that of your PC/Mac when you copy a file over.

You can choose either of those dates for sorting purposes - opting to sort "By date added to Library" or using the filesystem's date/time stamp.

Andre

[Arkats]

@Fitzian

1 hour ago, Fitzian said:

Ah! You are on a mac!

I was... 😉 Longtime ago. So which one will you me advise for PC?

Thank you very much for the kid3 manual I didn't have it!!

@andrewilley

49 minutes ago, andrewilley said:

PA doesn't know (or care) about your PC, Mac, or any other type of music collection you might have elsewhere.

I would say that he can know as soon as I synchronize my PC music folder with my Phone music folder, the files having a tag "Date Added to Library" from MusicBee. But I guess, he doesn't care...

But the very strange thing is why I have few albums which are in PA's library since a long time that appears as very recent (on the top of the list)? Knowing that I haven't done anything with these albums/files?!

1 hour ago, andrewilley said:

The filesystem's "last modified" timestamp is a different thing

Yes you're right. I had in mind that it might correspond to the moment you have added the file to your collection and fine-tuned some metadata. But then it is absurd because if you modify a metadata of on old file, the last modified date will change with no connection with the Date First Added to the Library...

[6b6561]

9 hours ago, Arkats said:

But the very strange thing is why I have few albums which are in PA's library since a long time that appears as very recent (on the top of the list)? Knowing that I haven't done anything with these albums/files?!

My guess would be that this is due to some past full rescan as older versions of Poweramp didn't retain the date added on a full rescan. Poweramp as of today is retaining the date added order.

[andrewilley]

9 hours ago, Arkats said:

the files having a tag "Date Added to Library" from MusicBee.

Nothing to do with Poweramp. Even if MusicBee is inserting that sort of information from its database into custom TXXX tags within audio files, Poweramp would not read or care about them anyway.

PA uses a fairly common and safe subset of the tags provided by the ID3 or Vorbis specs, and it does not read any custom tags that belong to other specific apps (MusicBee, MusicBrainz, etc).  It's hard enough to support an official spec, let alone whatever some other app has invented too.

 

"Date Added to Library" is the first time/date that Poweramp noticed that specific file in its scans - i.e. that exact directory path and filename, its contents are irrelevant - and added the song to its Library. If MusicBee has tweaked anything (folder path or name) then the file will be treated as a new item and scanned in again, with the newly scanned tags used to assign it to an Artist, Album, etc.

Andre

[6b6561]

13 hours ago, Arkats said:

Oh! Suddenly it seems quite austere to me.

The benefit of doing it "manually" is that you know exactly what happens and helps you to avoid any non-standard automagic things to take place that does creative things to your library. I must admit that I was looking for some tool to maintain the library earlier in the spring but I came to the conclusion that it wasn't worth the hassle.

I have a fairly automated workflow through which I import CD's to my library, rip using EAC to a Artist/Album folder, run a script that adds some tags like TCMP and TBPM. Run FolderSync on the Android device to add the tracks to Poweramp. Add the new album to iTunes on my PC and sync my iPhone.

(iTunes see's only a read-only copy of the library as iTunes was quite keen on applying it's own magic...)

[Arkats]

@6b6561

2 hours ago, 6b6561 said:

My guess would be that this is due to some past full rescan as older versions of Poweramp didn't retain the date added on a full rescan. Poweramp as of today is retaining the date added order.

Yes, it seems quite logic but I am not even sure it is the reason as I would have much more than few albums messed up.

In any case, I will  live with it as according to what Andre was written, it is feasible but complicated.

1 hour ago, 6b6561 said:

The benefit of doing it "manually" is that you know exactly what happens and helps you to avoid any non-standard automagic things to take place that does creative things to your library.

I guess it is wiser to keep as much control as you do.

@6b6561 & @andrewilley

1 hour ago, 6b6561 said:

to avoid any non-standard automagic things to take place that does creative things to your library.

That has been said several times when mentioning iTunes and MusicBee.

But what are we talking about exactly?

@andrewilley

2 hours ago, andrewilley said:

its contents are irrelevant

So, I guess that it allows us to modify a file (tags, even length) without it being considered as new when we do a rescan.