anyone can boot into that subsystem and access/alter the OS

Right. I was looking at it from a malware point of view, since OP’s question “is it safe” is vague and can have one of at least three meanings. If your phone is rooted and the wrong person has physical access to it, they can bypass factory reset protection, among other things

However, most phones nowadays encrypt their user’s data using the lock screen password as the encryption key. I think this would limit an attacker’s access to user data, but don’t quote me on that