No, there is nothing stopping you from running a baker on a remote server if you want to use a ledger. In fact, this can be a great way to manage your baking operations and ensure the security of your funds. You will need to make sure that the server is secure and that all of your baking operations are performed in a secure manner. Additionally, you will need to make sure that the server is running the latest version of the ledger software and that it is properly configured. With the right setup, you can be sure that your baking operations are secure and that your funds are safe.