Yes, it stays on. The X in the notification area doesn't close it. I have to open the app and close it, then it's gone.
Today I tried NOT connecting the USB, and just letting the bluetooth connect, and it didn't happen (i.e., the app was gone from the notification area).