Secure Your Secrets: Best Practices for Hardening HashiCorp Vault in Production
So, you’ve got Vault up and running, and you’re feeling pretty good about storing and managing secrets. But here’s the thing—running Vault in production is a whole different game. It’s not just about turning it on; it’s about hardening it to ensure that your Vault instance is secure, reliable, and resilient against attacks.
In this post, we’ll dive deep into production hardening techniques for Vault, covering encryption, mutual TLS (mTLS), firewall rules, and operational best practices. By the end, you’ll have the tools and knowledge to harden your Vault setup and run it like a pro in production.
Understanding Vault’s Security Model
Before diving into hardening steps, let’s examine Vault’s core security model, which is designed to ensure the highest standards of confidentiality, integrity, availability, accountability, and authentication.
• Confidentiality: Vault ensures confidentiality by encrypting all data at rest and in transit. This prevents unauthorized access and protects sensitive information from exposure.
• Integrity: Vault maintains data integrity through secure, cryptographic hashing and strict access control policies. This ensures that data has not been tampered with or altered by unauthorised users.
• Availability: Vault is designed for high availability, using a distributed architecture with clustering and disaster recovery features. These configurations help ensure that secrets and data are accessible to authorised users and systems without disruption.
• Accountability: Vault’s comprehensive audit logging allows for complete traceability of actions taken within the system. Every request is logged, enabling a clear record of who accessed what and when, making it easier to detect anomalies and meet compliance requirements.
• Authentication and Authorisation: Vault uses flexible authentication methods (like Userpass, AppRole, or LDAP all ultimately providing a token) and fine-grained policies to enforce strict access control. This framework ensures that only authorised entities can access or modify secrets, adhering to the principle of least privilege.
Now, while Vault’s out-of-the-box security features are robust, you can (and should) go further to harden your Vault instance in production using the some or all of the following techniques.
Key Hardening Techniques
TLS Configuration
When it comes to running Vault in production, securing data in transit is non-negotiable. Vault’s communication with clients involves sensitive information—secrets, keys, policies—so using a secure channel is crucial. Enter TLS (Transport Layer Security).
Vault defaults to enabling TLS unless you’re running in dev mode or set tls_disable (not recommended in production). You can integrate TLS with your organisation’s existing certificate authority, or, if needed, Vault can act as a certificate authority itself. Whichever option you choose, the important part is that data in transit stays encrypted.
Mutual TLS (mTLS) Authentication
mTLS goes a step further, requiring both the client and server to authenticate each other. This enforces client certificate validation, adding another layer of security and ensuring that only trusted clients can access Vault in a production environment.
Firewalling
TLS and mTLS alone aren’t enough. Restrict network access to Vault with a firewall or cloud security groups, limiting connections to trusted IP ranges or VPN networks. Vault should never be exposed to the public internet unless absolutely necessary. This perimeter defence is crucial for minimising potential entry points.
Operational Security
Vault is secure by design, but securing your production instance over time requires rigorous operational security practices:
• Use a Dedicated Service Account: Avoid running Vault as root or an administrator. Use an unprivileged account dedicated to Vault, reducing the risk of privilege-escalation attacks.
• Minimal Write Privileges: Limit the Vault service account to write only where necessary—local storage or audit log directories. The Vault binary and configuration files should remain unwritable.
• No Core Dumps or Swap: Disable core dumps and swap to avoid sensitive data being accessible in memory or on disk. For Linux users, setting RLIMIT_CORE=0 or LimitCORE=0 in the systemd service file will do the job.
• Single Tenancy: Vault works best on a machine without other processes, as this minimizes the risk of interference or attacks from other applications. When possible, prefer bare metal over VMs and VMs over containers.
• Firewall Rules: Use local firewalls or cloud security tools to limit Vault’s incoming and outgoing traffic to necessary services only, like NTP for time sync or database connections.
• Regular Updates and Patches: Keep your Vault instance updated with the latest security patches from HashiCorp. Staying on the latest release helps close known vulnerabilities.
• Least Privilege Access Control: Follow the principle of least privilege, giving users and applications only the access they need and nothing more.
• Secure Storage Backend: If you’re using an external storage backend (like Consul or etcd), configure it securely with its own TLS encryption and access controls to avoid becoming a weak link in the chain.
Additional Hardening Considerations
• Avoid Root Tokens: Root tokens should only be used during initial setup, then revoked. This ensures they don’t linger as high-value targets. For ongoing configuration, treat Vault policies and settings as code, with version control for better management.
• Short-Lived TTLs: When possible, configure short TTLs for issued credentials (tokens, certificates) to reduce exposure in case of compromise and minimise the need for revocation.
• Audit Logging: Enable audit logging to track all Vault operations, which provides a forensics trail if needed. Vault’s audit logs hash sensitive data, but restrict access to these logs to minimise exposure.
• Disable Shell Command History: For extra security, consider removing the vault command from shell history to avoid accidental leaks of sensitive commands.
• Configure User Lockout: Vault’s user lockout is enabled by default on approle, LDAP, and userpass methods. Ensure your settings align with your security policy to prevent brute-force attempts.
• No Clear Text Credentials: Never store credentials or HSM pins in clear text within the Vault configuration. Use environment variables where applicable, or platform-specific identity solutions like AWS Instance Profiles or Azure Managed Service Identities.
• Upgrade Frequently: Vault is under active development, and frequent updates bring not only new features but important security fixes. Stay subscribed to HashiCorp’s announcements to keep up with new releases and security patches.
Final Thoughts
Vault is a security-first tool, but production hardening is essential to ensure it remains resilient against attacks. By configuring TLS and mTLS, setting firewall rules, using least privilege access, and staying updated, you can build a fortified Vault deployment that’s ready for production challenges.
In addition to these measures, it's crucial to continuously monitor and audit your Vault environment to detect any anomalies or unauthorized access attempts. Implementing automated alerting systems can help you respond swiftly to potential threats.
By taking a proactive approach to security, you not only protect your secrets but also build trust with stakeholders who rely on the integrity and confidentiality of your systems.
Stay tuned for more as we dive into auto unsealing, and other advanced Vault topics in upcoming posts!
I hope this helps someone.
Cheers