A bastion host is a publicly accessible computer that has been hardened to protect the running services. A Linux-based bastion host is a good tool for allowing secure remote administration. The operating system is free and can be locked down without any special software.
Before installing Linux, inventory all of the servers and services that need to be accessed. Identify IP addresses and port numbers associated with each service. Be sure to identify the IP addresses of all of the clients that will need access. The bastion host is publicly accessible, but it does not need to be wide open for everyone.
The choice of Linux distribution for the the base is not really critical. The distribution is simply the starting point. A server oriented distribution is typically a better starting point since they have fewer running services to lock down. Choose a distribution that you are comfortable with and skills to support. Install the minimal or core packages, and remove any packages that are not needed. This reduces the attack surface area that needs to be defended.
Lock down the local host-based firewall before allowing any remote network access. Revisit the access inventory that was created earlier. Restrict both the inbound and outbound connections to just those that are necessary to get the job done. If you have a standalone firewall, the rule set for the bastion host should mirror the host-based rules. Determine whether you will access those via SSH or VPN tunnel. Use SSH keys or X509 certificates for authentication. Avoid simple password authentication, and to not tie this server to any single sign-on authentication service. This server should be a secondary layer of defenses for the administration tools. The system administrator must pass the bastion host and the access requirement of the administration tools independently. Test your firewall rules to make sure that they allow only the intended traffic.
What other services need to be running? None. A bastion host is not the proper place to be hosting a wiki or an SMTP gateway. Its sole purpose is to tunnel the remote access connection to the administration tool. After the server is built and locked down, install an intrusion detection system like OSSEC, Wazuh, or AIDE and make a backup of the configuration.
Once the bastion host is in production, keep it up to date with all current patches. Monitor the log files closely. A bastion host is a calculated risk, and the benefits of remote administration for system administrators in a pinch definitely outweighs the extra effort of watching another server.