Do you know gost aka go-simple-tunnel. It is a very powerful tunnel, it allow chain load the exit point. I once made an app for someone in UK to chain load gost client/server tunnel in wireguard. So that he can evade his ISP and has wireguard as exit point. The setup is like this, where as gost client convert wireguard to gost tunnel and send the gost server the destination wireguard server info, the gost server convert gost tunnel -> to wireguald. But the most interesting part is gost tunnel is any protocol you choose to evade your ISP.

Wireguad client -> gost clent -> gost server -> wireguard server.

I don't know how to call, if wireguard wrap gost, or gost wrap wireguard.

https://man.openbsd.org/vxlan.4#SECURITY seems unambiguous that it's intended for use in trusted environments (and all else being equal, I'd expect the openbsd man page authors to have reasonable opinions about network security), so it sounds like vxlan over ipsec/wg is probably the better route?

For site-so-site ovelay networks, use wireguard, vxlan should be inside of it, if at all. Your "network" is connected by wireguard, and it contains details like vxlan. Even within your network, when crossing security boundaries across untrusted channels, you can use wireguard.

Others mentioned tailscale, it's cool and all but you don't always need it.

As far as security, that's not even the consideration I had in mind, sure wireguard is secure, but that's not why you should have vxlan inside it, you should do so because that's the purpose of wireguard, to connect networks securely across security/trust boundaries. it doesn't even matter if the other protocol is also wireguard, or ssh or whatever, if it is an option, wireguard is always the outermost protocol, if not then ipsec, openvpn,softether,etc..whatever is your choice of secure overlay network protocol gets to be the tunnel protocol.

• p0w3n3d

  today at 6:43 AM

  There's also headscale - an OSS clone, if you're too paranoid to leave your keys to a company

Drop the VXLAN. There's almost never a good reason to stretch L2 over a WAN. Just route stuff across.

• dgl

  yesterday at 10:38 PM

  This.

  Instead you can create multiple Wireguard interfaces and use policy routing / ECMP / BGP / all the layer 3 tricks, that way you can achieve similar things to what vxlan could give you but at layer 3.

  There's a performance benefit to doing it this way too, in some testing I found the wireguard interface can be a bottleneck (there's various offload and multiple core support in Linux, but it still has some overhead).

• cjaackie

  yesterday at 10:31 PM

  This is the correct answer, routing between subnets is how it’s suppose to work. I think there are some edge cases like DR where it seems like stretching L2 might sound like a good idea, but it practice it gets messy fast.

  • formerly_proven

    yesterday at 10:39 PM

    VXLAN makes sense in the original application, which is to create routable virtual LANs within data centers.

• iscoelho

  yesterday at 11:03 PM

  EVPN/VXLAN fabrics are becoming industry standard for new deployments. MACSEC/IPsec is industry standard for site-to-site.

  You'd be surprised to know that this is especially popular in cloud! It's just abstracted away (:

  • wmf

    yesterday at 11:27 PM

    EVPN/VXLAN fabrics are becoming cargo culted. In most cases they aren't needed.

    • q3k

      yesterday at 11:44 PM

      Agreed. They've also been extremely finnicky from my experience - had cases where large EVPN deployments just blackholed some arbitrary destination MAC until GARPs were sent out of them.

      Also IME EVPN is mostly deployed/pushed when clueless app developers expect to have arbitrary L2 reachability across any two points in a (cross DC!) fabric [1], or when they want IP addresses that can follow them around the DC or other dumb shit that they just assumed they can do.

      [1] "What do you mean I can't just use UDP broadcast as a pub sub in my application? It works in the office, fix your network!" and the like.

      • iscoelho

        today at 12:00 AM

        VXLAN is used in cloud/virtualization networks commonly. VM HA/migration becomes trivial with VXLAN. It also replaces L3VPN/VRFs for private networks.

        • wmf

          today at 12:18 AM

          The good clouds don't support L2, they use a centralized control plane instead of brittle EVPN, and they virtualize in the hypervisor instead of in the switches. People are being sold EVPN as "we have cloud at home" and it's not really true.

          • iscoelho

            today at 8:59 AM

            AWS/GCE/Azure's network implementations pre-date EVPN and are proprietary to their cloud. EVPN is for on-premise. You don't exactly have the opportunity to use their implementation unless you are on their cloud, so I am not sure comparing the merits of either is productive.

    • iscoelho

      yesterday at 11:52 PM

      I don't disagree (:

      Though there are definitely use cases where it is needed, and it is way easier to implement earlier than later.

Not super related to the OP but since we're discussing network topologies; I've recently had an insane idea that nfs security sucks, nfs traversing firewalls sucks, kerberos really sucks, and that just wrapping it all in a wireguard pipe is way better.

How deranged would it be to have every nfs client establish a wireguard tunnel and only have nfs traffic go through the tunnel?

• throw0101c

  today at 12:03 PM

  > How deranged would it be to have every nfs client establish a wireguard tunnel and only have nfs traffic go through the tunnel?

  See perhaps NFS over TLS:

  * https://datatracker.ietf.org/doc/html/rfc9289

  * https://access.redhat.com/solutions/7079884

  * https://www.phoronix.com/news/Linux-6.4-NFSD-RPC-With-TLS

• QuantumNomad_

  yesterday at 11:20 PM

  > How deranged would it be to have every nfs client establish a wireguard tunnel and only have nfs traffic go through the tunnel?

  Sounds good to me. I have my Wireguard tunnel set up so that only traffic intended for hosts that are in the Wireguard network itself are routed over the Wireguard tunnel.

  I mostly use it to ssh into different machines. The Wireguard server runs on a VPS on the Internet, and I can connect to it from anywhere (except from networks that filter Wireguard traffic), and that way ssh into my machines at home while I am away from home. Whereas all other normal traffic to other places is unaffected by and unrelated to the tunnel. So for example if I bring my laptop to a coffee shop and I have Wireguard running and I browse the web with a web browser, all my web browsing traffic still gets sent the same normal way that it would even if I didn’t have the tunnel running.

  I rarely use NFS nor SMB, but if I wanted to connect either of those I would be able to that as well over this Wireguard setup I have.

• eugenekay

  yesterday at 11:18 PM

  I built a NFS3-over-OpenVPN network for a startup about a decade ago; it worked “okay” for transiting an untrusted internal cloud provider network and even over the internet to other datacenters, but ran into mount issues when the outer tunnels dropped a connection during a write. They ran out of money before it had to scale past a few dozen nodes.

  Nowadays I would recommend using NFS4+TLS or Gluster+TLS if you need filesystem semantics. Better still would be a proper S3-style or custom REST API that can handle the particulars of whatever strange problem lead to this architecture.

VXLAN over WireGuard is acceptable if you require a shared L2 boundary.

IPSec over VXLAN is what I recommend if you are doing 10G or above. There is a much higher performance ceiling than WireGuard with IPSec via hardware firewalls. WireGuard is comparatively quite slow performance-wise. Noting Tailscale, since it has been mentioned, has comparatively extremely slow performance.

edit: I'm noticing that a lot of the other replies in this thread are not from network engineers. Among network engineers WireGuard is not very popular due to performance & absence of vendor support. Among software engineers, it is very popular due to ease of use.

• Cyph0n

  today at 12:28 AM

  > Noting Tailscale, since it has been mentioned, has comparatively extremely slow performance.

  Isn't this mainly because Tailscale relies on userspace WG (wireguard-go)? I'd imagine the perf ceiling is much higher for kernel WG, which I believe is what Netbird uses.

  • iscoelho

    today at 8:17 AM

    wireguard-go is indeed very slow. For example, the official WireGuard Mac client uses it, and performance on my M1 Max is CPU capped at 200Mbps. The kernel WireGuard implementation available for Linux is certainly faster, but I would not consider it fast.

    Tailscale however, although it derives from WireGuard libraries and the protocol, is really not WireGuard at all- so comparing it is a bit apples to oranges. With that said, it is still entirely userspace and its performance is less than stellar.

    • Cyph0n

      today at 2:14 PM

      Well, according to this[1] bench, you can get ~10 Gbps with kernel WG.

      I'm interested in this because I'm working on a small hobby project to learn eBPF. The idea is to implement a "Tailscale-lite" that eliminates context switches by keeping both Wireguard and L3 and L4 policy handling in kernel space. To me, the bulk of Tailscale's overhead comes from the fact that the dataplane is running between user and kernel space.

      [1]: https://github.com/cyyself/wg-bench

• kosolam

  yesterday at 11:16 PM

  How is IPSec performance better than wg? I never heard this before, it sounds intriguing.

  • iscoelho

    yesterday at 11:34 PM

    At this time, there is no commercial offering for hardware/ASIC WireGuard implementations. The standard WireGuard implementation cannot reach 10G.

    The fastest I am aware of is VPP (open-source) & Intel QAT [1], which while it is achieves impressive numbers for large packets (70Gbps @ 512 / 200Gbps @ 1420 on a $20k+ MSRP server), is still not comparable with commercial IPsec offerings [2][3][4] that can achieve 800Gbps+ on a single gateway (and come with the added benefit of relying on a commercial product with support).

    [1] https://builders.intel.com/docs/networkbuilders/intel-qat-ac...

    [2] https://www.juniper.net/content/dam/www/assets/datasheets/us...

    [3] https://www.paloaltonetworks.com/apps/pan/public/downloadRes...

    [4] https://www.fortinet.com/content/dam/fortinet/assets/data-sh...

    • mlhpdx

      today at 3:51 PM

      This lack of ASIC is interesting to me. If it existed, that would very much change the game. And, given the simplicity of WG encryption it would be a comparatively small design (lower cost?)

    • iscoelho

      yesterday at 11:40 PM

      There are also solutions like Arista TunnelSec [1] that can achieve IPsec and VXLANsec at line-rate performance (21.6Tbps per chassis)! This is fairly new and fancy though.

      [1] https://www.arista.com/assets/data/pdf/Whitepapers/EVPN-Data...

  • hdgvhicv

    yesterday at 11:18 PM

    If you have an edge device which implements hardware IPsec at 10g+ but pushes WireGuard to software on an underpowered cpu then sure.

    • rebewhd

      yesterday at 11:33 PM

      While that's true, I'm not sure it's because of something inherent in IPsec vs WireGuard. It's more likely due to the fact that hardware accelerators have been designed to offload encryption routines that IPsec uses.

      One wonders what WG perf would look like if it could leverage the same hardware offload.

      • iscoelho

        yesterday at 11:35 PM

        Exactly this. I would love to see a commercial product with a hardware implementation for WireGuard, but it does not yet exist. IPsec, however, is well supported.

        • kosolam

          today at 10:47 AM

          Thanks for your answers. I wonder though, from the perspective of a small user that doesn’t have requirements for such bandwidth, how does ipsec compare with wg on other metrics/features? Is it worth looking into?

Whenever I see threads like this, I think its related but I'll be honest, my networking understanding might be limited.

I use Tinc as a daily driver (for personal things) and have yet to come up with a new equivalent, given that I probably should. Does Vxlan help here?

• iscoelho

  yesterday at 10:55 PM

  VXLAN is for L2 between campuses. It is commonly used in enterprise and cloud networks.

• imiric

  yesterday at 11:10 PM

  Tinc is a fantastic piece of software. Very easy to use and configure, and it just works.

  These days I lean towards WireGuard simply because it's built into Linux, but Tinc would be my second choice.

If a situation where production vxlan is going over Wireguard arises, then someone in leadership failed to plan and the underlying Wireguard tunnel is coping with that failure. No doubt, OP already knows this and all too well.

The problem is no doubt a people problem. I have learned to overcome these people problems by adhering to specific kinds of communication patterns (familiar to Staff Engineers and SVPs).

There is no reason that Wireguard over vxlan over Wireguard can't work, even with another layer (TLS) on top of Wireguard. Nonetheless it is very suboptimal and proprietary implementations of vxlan tend to behave poorly in unexpected conditions.

We should remember that vxlan is next-get vlan.

The type of Wireguard traffic encapsulated within the vxlan that comes to mind first is Kubernetes intra/inter-cluster pod-to-pod traffic. But this Wireguard traffic could be between two legacy style VMs.

If I were the operator told "you need to securely tunnel this vxlan traffic between two sites" I would reach for IPsec instead of Wireguard in an attempt to not lower the MTU of encapsulated packets too much. Wireguard is a layer 4 (udp) protocol intended to encapsulate layer 3 (ipv6 and legacy ip) packets.

If I were the owner of the application I would bake mutual TLS authentication on QUIC with "encrypted hello" (both elliptic and PQ redundant) into the application. The applications would be implemented in Rust, or if not practicable to implement the application in Rust I would write into the Helm chart a sidecar that does such a mutual TLS auth part (in Rust of course).

I would also aggressively "ping" in some manner through the innermost encapsulation layer. If I had tenancy on a classic VM doing Wireguard over the vxlan I would have "ping -i 2 $remote_inside_tunnel_ipv6" running indefinitely.

I use vxlan on top of wireguard in my hobby set up. Probably wouldn't recommend it for an actual production use-case. But that is more or less because of how my homelab is setup (Hetzner -> Home about 20ms latency roundtrip).

I considered dropping my root wireguard and setting up just vxlan and flannel, but as I need NAT hole punching I kind of need the wireguard root so that is why i ended up with it.

Going Wireguard inside the vxlan (flannel) in my case, would likely be overkill, unless I wanted my traffic between nodes between regions to be separated from other peers on the network, not sure where that would be useful. It is an easy way of blocking out a peer however, but that could just as well be solved on the "root" wireguard node.

There might be some MTU things that would be messed up going nested wireguard networks.

> Let’s agree going recursive (WireGuard inside VXLAN inside WireGuard) is a bad idea.

But it's not necessarily a bad idea. It depends on the circumstances, even when traversing a public network.

Is there a WireGuard equivalent that does L2 instead of L3? Need this for a virtual mesh network for homelabbing. I have this exact setup, running VXLAN or GENEVE over WireGuard tunnel using KubeSpan from Talos Linux but I simply think having L2 access would make load balancer much easier

• kjuulh

  yesterday at 10:28 PM

  You can see my reply below: https://news.ycombinator.com/item?id=46609044 I believe our setups are pretty equivalent.

  I achieve load balancing by running native wireguard on a vps at hetzner, I've got a native wireguard mesh, I believe Talos can do the same, where the peers are manually set up, or via. tailscale etc. I then tell k3s that it should use the wireguard interface for vxlan, and boom my kubernetes mesh is now connected.

  flannel-iface: "wg0" # Talos might have something similar.

  I do use some node-labels and affinities to make sure the right pods end up in the right spot. For example the metallb annoucer always has to come from the hetzner node. As mentioned in my reply below, it takes about 20ms roundtrip back to my homelab, so my sites can take a bit of time to load, but it works pretty well otherwise, sort of similar to how cloudflare tunnels would work, except not as polished.

  My setup is here if it is of help

  https://git.kjuulh.io/kjuulh/clank-homelab-flux/src/branch/m...

• dietr1ch

  yesterday at 10:30 PM

  Is this your use case?

  https://docs.zerotier.com/bridging/

  • stevefan1999

    today at 2:20 AM

    I used to like ZT but they went BSL. Plus it is not running in kernel unlike WireGuard. Memory usage is extremely high.

    I used to run my K8S homelab through ZT as well. Latency is extremely bad.

    What I wanted is more like meshed L2TPv3, but L2TPv3 is extremely hard to setup nowadays

• viraptor

  yesterday at 10:32 PM

  ZeroTier does L2.

What are your discovery mechanisms? I don't know what exists for automatic peer management with wg. If you're doing bgp evpn for vxlan endpoint discovery then I'd think WG over vxlan would be the easier to manage option.

• uberduper

  yesterday at 10:22 PM

  If you actually want to use vxlan ids to isolate l2 domains, like if you want multiple hypervisors separated by public networks to run groups of VMs on distinct l2 domains, then vxlan over WG seems like the way to go.

What problem is being solved here?

What are you trying to do? Why are you trying to link networks across the public internet?

For traversing public networks, simply consider BGP over Wireguard. VXLAN is not worth it.

• ghxst

  yesterday at 10:18 PM

  I've used wireguard for a while, not sure why I never considered doing BGP over it, might make for a fun weekend project.

  • tucnak

    yesterday at 10:26 PM

    BGP is vastly superior to any L2 make-believe trash you can imagine, and amazingly, it often has better hardware offloading support for forwarding and firewalls. For example, 100G switches (L3+) like MikroTik's CRS504 do not support IPv6 in hardware for VXLAN-encapsulated flows, but everything just works if you choose to go the BGP route.

    L2 is a total waste of time.

    • iscoelho

      yesterday at 11:14 PM

      Any ASIC switch released in the last decade from Cisco/Juniper/Arista supports EVPN/VXLAN in hardware. EVPN is built on BGP. This has become the industry standard for new enterprise and cloud deployments.

      The lack of support for hardware EVPN is one of the many reasons that Mikrotik is not considered for professional deployments.

      • hdgvhicv

        yesterday at 11:22 PM

        Mikrotik is used for professional deployments all over the world. Right tool for the right job.

        People who think one size fits all are not professional.

        • iscoelho

          yesterday at 11:49 PM

          If I can source an enterprise Cisco/Juniper/Arista ASIC switch that is 1) rock-solid 2) full featured 3) cheaper - which I can - there is unfortunately no rationale where Mikrotik would be applicable in any professional project of mine.

          With that said, I love Mikrotik for what it is: it is very approachable and it fills a niche. I believe it has added a lot of value to the industry and I'm excited to see their products mature.

          • hdgvhicv

            today at 7:37 AM

            Based on the lldp messages I see across dozens of countries, the majority of business isps globally use mikrotiks at their edge.

            • iscoelho

              today at 8:38 AM

              I'm curious what you classify as a business ISP?

              Take a look at AMS-IX, one of the largest internet exchanges: https://bgp.tools/ixp/AMS-IX

              21/1020 (2%) of all peers are Mikrotik. 15 (1.4%) of those are >=1000mbps. 7 (0.6%) of those are 10gbps. None are larger than 10gbps.

              • hdgvhicv

                today at 10:30 AM

                Those selling end services to businesses.

                I have a mix of equipment from heavyweight juniper mxs at peeing points to arista dcs/ccs in large sites to £50 mikrotiks in the smallest branch offices.

                Right tool for the right job, mikrotik is often but not always the right tool.

              • tucnak

                today at 10:59 AM

                You're referencing backbone, not edge. It has only been a few years that MikroTik had offered a 100G solution, let alone became competitive in it. You won't find it in the backbone yet. However, many European ISP's have largely upgraded their distro and aggregation switches to MikroTik over the last five years. There's a sovereignty push, too. I would guess edge is similar, but there's too many cheap options there so probably not that much.

                If your impression is based on data circa ~2020, you should re-evaluate your priors with the recent packages in mind. See https://mikrotik.com/product/crs812_ddq

          • tucnak

            today at 7:09 AM

            You're delusional on price. I wouldn't touch severely overpriced and backdoored American switches with a 10-foot pole! Meanwhile, MikroTik just released a 400G switch in under two grand. To buy Cisco/Juniper/Arista with your own money in 2025 you have to be super rich and super stupid. And I say this as a guy that buys 100G stuff from Xilinx.

            • iscoelho

              today at 8:09 AM

              I have not seen a case where I could not source a Juniper switch (for example) for lower $/port than Mikrotik, even at 400GE. It is unheard of to pay MSRP. YMMV.

Tell us why you think so at least.

• ronsor

  yesterday at 9:35 PM

  Reduced MTU chopping off your maximum packet size from all the extra headers and other overhead you're adding?

WG is L3 transport

VXLAN is L2-like tranport over L3

You can have EoIP over WG with any VLANs you like.

You can have a VXLAN over plain IP, over EoIP, over WG, over IPSec. Only WG and IPSec (with not NULL sec) do providecany semblance ofvencryption in transit

And mandatory X\Y problem.

I mean, ultimately, thats how Google routes internally.

IPSec-equivalent, VXLAN-equivalent, IPSec-equivalent.

Prevents any compromised layer from knowing too much about the traffic.

• pixl97

  yesterday at 9:45 PM

  Internal is fine because you control things like MTU so you don't have to worry about packet fragmentation/partial loss.

• als0

  yesterday at 10:03 PM

  That seems like an awful amount of overhead for questionable gain.

  • _bernd

    yesterday at 10:07 PM

    Links between, and in between data centers use so called jumbo frames with an mtu of over 9000. Not joking.

    • xoa

      today at 12:57 AM

      Worth mentioning that links at home can use them too, jumbo frame support was rare at one point but now you can get them on really cheap basic switches if you're looking for it. Even incredibly cheap $30 (literally, that's what a 5 port UniFi flex mini lists for direct) switches support them now. Not just an exotic thing for data centers anymore, and it can cut down on overhead within a LAN particularly as you get into 10/25/40/100 Gbps stuff to your own NAS/SAN or whatever.

• tucnak

  yesterday at 10:23 PM

  What gave you that idea? Internally, Google uses GRE/GENEVE-like stuff but for reasons that have nothing to do with "preventing compromise" or whatever, but because they're carrying metadata (traces, latency budgets, billing ids.) That is to say, encapsulation is just transport. It's pretty much L3 semantics all the way down... In fact, this is more or less the point: L2 is intractable at scale, as broadcast/multicast doesn't work. However, it's hard to find comparisons to anything you're familiar with at Google scale. They have a myriad of proprietary solutions and custom protocols for routing, even though it's all L3 semantics. To learn more:

  Andromeda https://research.google/pubs/andromeda-performance-isolation...

  Orion https://research.google/pubs/orion-googles-software-defined-...

  • zeroxfe

    today at 12:51 AM

    The last time I was there, there were many layers of encap, including MPLS, GRE, PSP, with very tightly managed MTU. Traffic engineering was mostly SDN-managed L3, but holy hell was it complex. Considering that Google (at the time) carried more traffic than the rest of the Internet combined, maybe it was worth it.

  • DiabloD3

    yesterday at 11:02 PM

    What gave me that idea? Talks and research papers from Google network engineers over the past decade.

    • tucnak

      today at 11:04 AM

      Where are you getting at VXLAN-equiv, IPsec-equiv, etc. specifically? ALTS/PSP is not "IPsec-equivalent"

vxlan inception is fun ! said no one ever !

Not sure I understand, but why not Tailscale?

• sy26

  yesterday at 10:37 PM

  In my case, Tailscale does not implement K8S CNI.