Update: Seems like my main issue was solved by turning off the no reshuffle on large lists. However the issue of me searching a song and it instead playing the first song in the order instead of the selected song is an issue when "no reshuffle large lists" is on. Steps on how to trigger below.
No reshuffle large lists on.
Start a shuffled playlist or all songs or rated songs.
CHANGE the shuffled type. For me default is songs/categories. In this example I changed it to All.
Search for a song and click to add it. Instead of it playing the song it just goes back to the first song in the order.
Is this intended? I have turned this setting off anyways but figured I'd mention it. The wording on the settings seems like this is unintentional