Site-to-Site VPN
On-Premises
strongSwan
FRR
Route 53 Resolver
Aurora
PostgreSQL
Terraform
KMSThe Problem
A real on-premises application needs to use a managed database hosted in AWS — an Aurora PostgreSQL cluster — securely: the database must never be exposed to the public internet, and the on-prem host must not hold long-lived AWS or database credentials. This is the everyday shape of hybrid integration. And hybrid traffic is rarely one-way: sooner or later something in the cloud needs data that still lives on-prem, so the design has to carry the reverse path too.
The naive answer — point the app at a public RDS endpoint with a static database password over TLS — fails on two axes an architect cares about. Network: the database is reachable from the public internet, exactly the exposure a security review will flag. Identity:a long-lived password and AWS access key sitting on an on-prem box are standing secrets — copyable from a config dump, a backup, or an ex-employee’s notes — whose rotation is a manual, therefore rare, event.
The architecture in this post composes a small set of controls instead. An IPsec Site-to-Site VPN — two tunnels, BGP running over both, active/active — provides a private, self-healing path into a fully-private VPC: no internet gateway, no NAT; the tunnels are the only way in, VPC interface endpoints the only way out to AWS APIs. Aurora is reached over that path via its private network interface. Credentials are vended at runtime: the only durable secret on-prem is a bootstrap principal that can do nothing but sts:AssumeRole a tightly-scoped role, which reads exactly one secret or mints a 15-minute RDS IAM auth token. And because no private name resolves across the boundary by default, hybrid DNS — Route 53 Resolver inbound and outbound endpoints — is first-class infrastructure, not an afterthought.
The lab deliberately departs from the reference tutorials in three ways. It uses a real, independent on-premises server (Rocky Linux 9, its own public IP, outside AWS) running strongSwan as a software Customer Gateway with FRR speaking BGP — not a second VPC pretending to be an office. It targets a stateful database session, which exercises routing, cross-VPN DNS, and security-group depth that an S3 API call never touches. And it is built to production posture — both tunnels genuinely active, pinned crypto, audit alarms on — with the few lab-shaped shortcuts documented as accepted risks rather than hand-waved.
Everything AWS-side is Terraform in ap-southeast-1; everything on-prem is one Linux server configured from a runbook. What follows is the architecture, the live evidence, and the decisions — including the two failures that reshaped the design.
Architecture & Data Flow
The whole system in plain words first. Imagine an office and a private cloud connected by two redundant encrypted pipes. Neither side is reachable from the internet — the pipes are the only way in. Each side runs a small phone bookso it can look up the other side’s names. When a program in the office wants the cloud database, it first trades its low-value key for a short-lived pass, uses the pass to fetch what it needs, and the pass expires minutes later. When a program in the cloud wants the office database, it looks the name up through its phone book and connects straight through the pipe. If one pipe is cut, traffic moves to the other within seconds — nobody reconfigures anything.
Concept — BGP in one paragraph
10.100.0.0/24”) and learnsthe other’s. Liveness is a heartbeat: a keepalive every 10 seconds, a hold time of 30 — “if I hear nothing for 30 seconds, I declare you dead, withdraw your routes, and send traffic another way.” That last sentence is the entire high-availability story of this design.① The bridge — two tunnels, BGP over both
strongSwan negotiates two IKEv2 tunnels (crypto pinned to AES-256 / SHA-256 / DH group 20 on both ends) to AWS’s two tunnel endpoints. Each tunnel is route-based: its traffic selectors are wide open and the tunnel is bound by an if_id tag to a plain Linux XFRM interface — a virtual network card that routes can point at. Each tunnel also carries a tiny /30“inside” network (link-local 169.254.x.x) whose only job is to host a BGP session — an intercom line drilled through the pipe. FRR advertises the on-prem LAN and learns the VPC range over bothtunnels; the kernel installs the best path, and if that tunnel dies, BGP’s 30-second hold time withdraws it — traffic moves to the other tunnel with no human action. On the AWS side, VGW route propagation injects the learned on-prem route into the VPC route tables automatically.
② ⑥ Hybrid DNS — the two phone books
The VPC’s built-in resolver is not reachable across a Site-to-Site VPN (an AWS restriction) — that single fact is why the two Route 53 Resolver endpoints exist. On-prem → AWS: dnsmasq forwards exactly three AWS zones (RDS, Secrets Manager, STS) through the tunnel to the Resolver inbound endpoint, so the Aurora endpoint name resolves to its private IP from outside AWS. AWS → on-prem: the Resolver outbound endpoint plus a forwarding rule send any *.onprem.lab query from inside the VPC to the same dnsmasq. Both directions, one pair of phone books — and the endpoint IPs are pinned in Terraform so the on-prem forwarder config can never silently go stale.
③ ④ ⑤ Credential vending — on-prem reads Aurora, no stored password
The on-prem CLI profile calls AssumeRole; the call travels through the tunnel to the STS interface endpoint(an ENI in the private subnet — AWS’s public API brought inside the VPC). The role’s trust policy carries an aws:SourceVpce condition — only callers arriving through this exact endpoint are accepted, so a stolen bootstrap key used from anywhere else on the internet is refused. With the resulting 1-hour temporary credentials the host can do exactly two things: read one secret via the Secrets Manager endpoint, or mint a 15-minute RDS IAM auth token that Aurora accepts in place of a password. psqlthen connects to Aurora’s private endpoint with TLS enforced end-to-end: sslmode=verify-fullon the client (check the certificate’s name, not just that it has one), rds.force_ssl=1 on the server.
⑥ ⑦ The reverse path — AWS reads the on-prem database
An EC2 “initiator” inside the VPC — no SSH key, no inbound ports, reachable only through SSM Session Manager via interface endpoints — resolves db.onprem.lab through the outbound resolver and opens a psql session across the tunnel to PostgreSQL 16 on the on-premises server. That server’s client-access config allows exactly one remote rule: one database, one user, only from the VPC range, with scram-sha-256authentication. The result: each side consumes the other’s database over a path that never touches the internet.
Results & Evidence
The screenshots below show the system running end-to-end in a live AWS account (ap-southeast-1) and on the on-premises server itself. Each view validates a specific claim from the Architecture section above.
The bridge — two tunnels, both alive
The money shot for the whole design: the VPN connection’s Tunnel details tab with both tunnels UP. AWS provisions the two tunnels on independent endpoints so it can perform maintenance on one at a time — a design you only benefit from if your side actually keeps both alive.
The on-prem view of the same fact. swanctl --list-sas shows two ESTABLISHED IKE security associations — one per AWS endpoint — each with the pinned AES_CBC-256/HMAC_SHA2_256 proposals and its own child SA carrying traffic.
And the routing layer on top: FRR’s BGP summary shows two neighbors in Established state — the VGW inside addresses across tunnel 1 and tunnel 2 — each having learned exactly one prefix (PfxRcd=1 — the VPC range) while advertising the on-prem LAN back to AWS.
The AWS side proves BGP did real work: the private route table’s route to the on-prem network shows Propagated = Yes — nobody typed that route; the VGW injected it from what BGP learned.
Failover — kill a tunnel, keep the sessions
To prove the HA story, tunnel 2 was deliberately held down. The CloudWatch TunnelState alarm graph tells the whole timeline: the dip is the outage window (metric drops below 1, alarm goes to ALARM), during which BGP withdrew the dead path within its 30-second hold time and traffic flipped to tunnel 1 with no human intervention. When the tunnel was re-initiated, the metric recovered and the alarm returned to OK — visible proof of both the failure and the hands-off recovery.
Hybrid DNS — private names across the boundary
From the on-premises server, one dig answers both directions of the DNS story: the Aurora cluster endpoint resolves to its private VPC address (something no public resolver can produce), and db.onprem.lab resolves locally — the same name the cloud side will ask for through the outbound endpoint.
The machinery behind that answer: the Resolver inbound endpoint with one ENI per AZ, IPs pinned in Terraform (192.168.0.10 / 192.168.1.10) so the on-prem forwarder config never goes stale.
Credential vending — Aurora with no stored password
The trust policy is where the vending chain gets its teeth. The access role trusts only the bootstrap user, and only when the call arrives through the lab’s own STS endpoint — the aws:SourceVpce condition. A leaked bootstrap key replayed from anywhere else fails this check before IAM even evaluates permissions.
The vend itself, end to end in one terminal: assume the role, mint a 15-minute RDS IAM token (piped straight into PGPASSWORD — never printed), and connect with sslmode=verify-full. The query returns current_user = app_iam from the Aurora writer’s private address — a database session with no password stored anywhere on the host.
And the audit trail: CloudTrail records the AssumeRole event with source IP 10.100.0.1 — the on-prem LAN address, proof the call rode the tunnel into the VPC endpoint rather than the public internet. This exact event pattern (non-service principals assuming this role) is what the vending alarm is scoped to.
The reverse path — cloud reads the office database
The reverse-path traffic source is deliberately boring: an EC2 instance launched with no key pair at alland no inbound security-group rules. In a VPC whose whole point is “no ingress but the tunnel”, the operator door is SSM Session Manager — not SSH.
Through that SSM session, psql -h db.onprem.lab resolves via the outbound resolver and lands on the on-prem PostgreSQL: inet_server_addr() = 10.100.0.1 — the cloud side reading the office database across the same two tunnels, in the opposite direction.
Guardrails on
Steady state: the whole alarm fleet — two tunnel-state alarms, Aurora CPU, and the four CloudTrail metric-filter alarms (credential vending, secret reads, root usage, trail tampering) — sitting green. During the failover demo and every vend, the relevant alarm fired and emailed; the rest never false-alarmed, because the vending filters are scoped to non-service principals hitting this role, not AWS housekeeping noise.
Design Decisions
A real on-premises server, not a second VPC
A second VPC is simpler — one console, one firewall model, trivial teardown — but it never exercises what actually breaks real hybrid VPNs: provider network policy, NAT traversal, an external firewall, no single-pane visibility. The real server is precisely what surfaced the two hardest lessons below.
Concept — policy-based vs route-based IPsec
tcpdump. A route-based tunnel turns the tunnel into what looks like an ordinary network cable: a virtual interface (on Linux, an XFRM interface) that the routing table — including routes a BGP daemon installs — can point at. Selectors go wide open because routing, not IPsec policy, now decides what enters the tunnel.Route-based (XFRM) + BGP, not policy-based + static
The lab started policy-based/static and migrated. The forcing fact: policy-based IPsec cannot run two tunnels with identical selectors — the second 0.0.0.0/0 connection is rejected, capping you at one active tunnel, i.e. guaranteed downtime whenever AWS maintains that tunnel endpoint. Route-based tunnels move the “what goes in” decision to the routing table, which is what lets BGP run over both tunnels and deliver ~30-second automatic failover. Static routing cannot express “prefer whichever path is alive.”
strongSwan, not Libreswan — a capability decision
Libreswan 4.x failed route-based HA on this exact design twice: with XFRM interfaces it refuses the second wide-open connection (route conflict); with VTI, inbound traffic requires a packet mark that wire packets don’t carry — everything decrypts, then silently drops. strongSwan’s per-connection if_id binding (each tunnel tagged to its own interface) avoids both, and strongSwan is AWS’s documented customer-gateway software for this design.
Verify the hosting network passes raw ESP — before you build
ESP is IPsec’s data protocol (IP protocol 50 — no ports), and some hosting networks silently drop it. The first hosting location did exactly that: IKE completed, AWS showed “IPSEC IS UP”, and not one decrypted packet ever arrived — every dashboard green, nothing working. The fix was moving the server to a host that passes ESP cleanly. Promote this to a pre-check: probe ESP reachability before trusting any site with an IPsec workload.
Hybrid DNS via Resolver endpoints, not endpoint-URL overrides
Hard-coding endpoint URLs and hosts-file entries avoids standing infrastructure, but it doesn’t survive contact with strict TLS (verify-full needs real names), RDS token minting (the signer embeds the hostname), or the reverse path. Resolver inbound + outbound endpoints with Terraform-pinned ENI IPs are the production answer — at an honest price (see Cost: this is the most expensive line in the whole lab).
Aurora over the VPN needs routing, not PrivateLink
A managed database’s data plane is just a private IP in your subnet — route to it over the tunnel like any other host. PrivateLink interface endpoints are for AWS service APIs, which is why STS and Secrets Manager get endpoints and Aurora does not. Conflating the two is a common design error.
Fence the trust policy with aws:SourceVpce
The access role only accepts AssumeRole calls arriving through the lab’s own STS endpoint — the right key at the wrong door is refused. It converts a credential control into a network control: even a leaked bootstrap key is useless from outside the tunnel. Cheap to add, disproportionate value.
RDS IAM auth tokens over fetched passwords
Reading the master secret proves the vending chain, but the cleaner end state — the primary path here — is rds-db:connect: exchange the IAM identity for a 15-minute signed token; no database password exists on-prem at all. One sharp edge: the IAM resource ARN uses a colon before the DB user (dbuser:app_iam) — a slash produces not an IAM error but a generic PAM authentication failure. Hours of misdirection.
Concept — MTU, MSS, and the black hole
MSS clamp + source-pinning — the two lines that carry the system
Tunnel interfaces run at MTU 1400, so TCP MSS is clamped to 1360 — without it, even BGP itself flapped once its route updates exceeded one packet. And because BGP speaks over link-local /30 addresses, Linux would stamp on-prem→VPC packets with a 169.254 source AWS can’t answer; a one-line route-map pins the source to the LAN IP instead. Neither line appears in any quickstart. Both are mandatory.
SSM-only access for the cloud-side initiator
No key pair, no inbound rules, no internet path — Session Manager via the SSM interface-endpoint trio is the only door, and every session keystroke streams to an encrypted log group. In a VPC whose entire point is “no ingress but the tunnel”, opening SSH — even from the VPN — would undercut the design.
Extended Knowledge
The two-phase IKE handshake, in plain terms
Before any data flows, the two ends must (1) prove who they are and agree on a private management channel, then (2) agree on the keys that encrypt the actual traffic. That two-step is IKE (Internet Key Exchange); its output is a pair of Security Associations — one-way “contracts” describing exactly how each direction of traffic is encrypted. Phase 1 authenticates (pre-shared key here) via a Diffie-Hellman exchange — mathematics that lets two parties agree on a shared secret while never transmitting it. Phase 2 negotiates the data-plane contracts ESP uses. AWS offers a broad menu of legacy-compatible proposals; pin the strong end (IKEv2, AES-256, SHA-256, DH 20, DPD) in IaC and mirror it exactly on the gateway — a proposal mismatch is a negotiation failure by design.
ESP, NAT-T, and the provider that eats protocol 50
Concept — ESP and NAT-T
The real-world catch is different: some hosting networks silently drop protocol-50 packets altogether — and that failure looks exactly like a healthy tunnel that carries no traffic. IKE (UDP) completes, status pages go green, and nothing forwards. It cost this lab a full server migration to learn: the hosting network’s protocol policy is part of your architecture.
What the /30 “inside” links are for
Each AWS tunnel comes with a tiny two-host network from the link-local range. It carries no user data — it exists purely so each end of the tunnel has an IP address the other can talk BGP to. One subtle interplay: BGP’s route updates are TCP, so an unclamped MSS can make the routing protocol itself flap while small pings still pass — a false-healthy state that is thoroughly misleading to debug.
Interface vs gateway endpoints — and who can reach them
Both keep AWS API traffic off the internet; their reachability differs decisively. A gateway endpoint (S3, DynamoDB) is a route-table shortcut — free, but only usable from inside the VPC, so it can never serve on-prem callers over a VPN. An interface endpoint is an ENI with a private IP in your subnet — billed hourly, but reachable by anything that can route to it, including on-prem. That is why STS and Secrets Manager are interface endpoints here, while a free S3 gateway endpoint lets the no-internet EC2 instance reach OS package repos.
Private DNS across the boundary — both directions
Private hosted zones and endpoint DNS names answer only resolvers inside the VPC, and the VPC’s native resolver is unreachable over Site-to-Site VPN. The inbound endpoint exists precisely for outside forwarders; the mirror problem takes the outbound endpoint + a forwarding rule. Two production details from this build: pin the resolver ENI IPs in IaC so the on-prem forwarder cannot silently go stale, and firewall the on-prem DNS service to accept queries only from the outbound endpoint ENIs — a DNS server reachable from the whole tunnel is unnecessary surface.
Throughput ceilings — when a Transit Gateway earns its cost
Each VPN tunnel caps at ~1.25 Gbps, and a VGW’s two BGP tunnels are active/active for availability, not added bandwidth. Exceeding the per-tunnel ceiling, or fanning one VPN out to many VPCs, is the signal to move to a Transit Gateway (ECMP across tunnels). Below that, the VGW is the cheaper, simpler choice — and with BGP it already delivers the failover story.
Security & Cost Posture
Security highlights
- Encryption in depth — pinned IPsec on the wire; TLS end-to-end for the database (
rds.force_ssl=1server-side,verify-fullclient-side); KMS at rest for Aurora, secrets, logs, and Terraform state - Identity short-lived and fenced — assume-only bootstrap principal, one-secret access role,
aws:SourceVpcetrust condition, 15-minute DB tokens instead of passwords - Structural network isolation — no IGW, no NAT, no public subnets; Aurora admits 5432 only from the on-prem range; on-prem firewall default-deny with three narrow allows
- Watched, not just walled — CloudTrail metric alarms scoped to real vending events, secret reads, root usage, trail tampering; tunnel + DB alarms; Flow Logs, Resolver query logs, SSM session logs → one SNS topic
- Residual risks named, not hidden — PSKs and the bootstrap key exist in Terraform state (state bucket treated as sensitive even after teardown); AWS-managed KMS keys; single-AZ Aurora — each accepted for a teardown-friendly lab with its production precondition documented
The cost surprise
Verified against the AWS Pricing API and the live deployed state — not list-price guesses. Running 24×7 this design costs ~$18.85/day ≈ $573/mo, and the ranking surprises most people: the VPN is nearly the cheapest line.
| Line item | Config | Est./month |
|---|---|---|
| Route 53 Resolver endpoints | inbound + outbound, 4 ENIs (AWS minimum) | ~$365 |
| VPC interface endpoints | 8 AZ-endpoints (STS, SM, SSM trio, Logs) | ~$76 |
| Aurora PostgreSQL | 1× db.t4g.medium, <1 GB storage | ~$85 |
| Site-to-Site VPN | 1 connection, both tunnels included | ~$36.50 |
| EC2 initiator | t4g.micro + 8 GiB gp3 | ~$8.50 |
| CloudWatch / KMS / SNS / logs | misc | ~$3 |
| Total | ~$18.85/day | ~$573 |
Two structural lessons. Transparent two-way DNS is the most expensive convenience in a hybrid design — the Resolver endpoints alone are ~64% of the bill. And the top two lines are AWS architectural minimums, so the only real lever is teardown discipline — the design is deliberately teardown-friendly (no deletion protection, no final snapshot, zero-day secret recovery).
Takeaways
- The hosting network is part of the design. Probe raw ESP (IPsec’s port-less data protocol) before trusting any site with an IPsec workload — a provider that silently drops it produces green dashboards and zero traffic.
- Policy-based IPsec caps you at one active tunnel. Route-based tunnels (each one a routable interface) with BGP over both turn AWS’s “two tunnels” into ~30-second hands-off failover.
- The IPsec software choice is a capability decision. strongSwan’s per-tunnel
if_idinterface binding makes two parallel route-based tunnels work where Libreswan’s mechanisms both fail. - A managed database needs routing, not PrivateLink. Aurora is already a private IP in your subnet; only the AWS service APIs (STS first, then Secrets Manager) need interface endpoints — and every hop must be private or the chain fails.
- Hybrid DNS is mandatory machinery — and the biggest line item. The VPC’s native resolver is unreachable over the VPN, strict TLS and token minting need real hostnames, and the Resolver endpoints that solve it cost more than Aurora and the VPN combined. Budget DNS as a first-class component.
- Two lines of config carry the whole system. An MSS clamp (without it, even BGP flaps — a green-dashboards-nothing-works failure) and source-pinning (always stamp outgoing traffic with a return address the other side can route back to).
- Credential vending removes the standing secret. Assume-only key +
aws:SourceVpcefence + 15-minute RDS IAM tokens = no database password and no usable long-lived AWS key on the host, with every vend audited and alarmed. - Everything here bills by the hour whether traffic flows or not. For a lab, teardown beats every other cost optimization.
References
- Set up Site-to-Site VPN PoC — CloudMentor (Part 1)
- AWS Site-to-Site VPN → S3 access — CloudMentor (Part 2; this post adapts the VPN pattern to a private Aurora workload)
- AWS Docs — Site-to-Site VPN customer gateway devices & routing options
- strongSwan documentation — swanctl.conf, XFRM interfaces (if_id)
- FRRouting documentation — BGP
- Infrastructure repo (Terraform, living documentation, and the full on-premises runbook) — private