Complete Guide for VPN Protocols

James Milin-Ashmore

James Milin-Ashmore Last updated: November 10, 2021

Complete Guide for VPN Protocols

Disclaimer: Partnerships & affiliate links help us create better content. Learn how.

The average VPN has at least a couple of VPN protocols on offer, but what’s the difference between the likes of OpenVPN and IPsec, and will there be an impact on performance for the average user?

The protocol selected will determine how your data travels from your device to a VPN server, with various configurations that focus on speed, privacy, and security. For example, it would be better to pick a protocol that emphasizes quick connections if you just want to unblock Netflix

Here’s a brief introduction to the different types of VPN protocols available, including what to expect from each one. 

What Is a VPN Protocol?

VPN protocol diagram

VPNs create a secure point-to-point connection between your device and their servers for the purpose of added privacy and anonymity. Many will use a tunneling protocol, which essentially determines how the data is sent to and from your device.

A VPN protocol is a set of instructions that are used to transmit your online traffic safely, often while assigning the user with a new IP address. There are many protocols that have been released over the years, from famous open-source options to proprietary tech built for a specific service. 

The type of protocol used will have an impact on aspects such as speed and security. You can guess that this is especially important for the end-user. (After all, you wouldn’t want to select a protocol with poor encryption methods or slow connection speeds if you’re using the VPN for streaming content.)

To make things easier to digest, we’ve listed the most common VPN protocols below along with pros and cons.

OpenVPN

As protocols go, it’s probably a good idea to start with OpenVPN. It was created and released in 2001 by James Yonam, who is the current CTO of OpenVPN Inc. It earns the name due to being open-source, and it’s available on all major platforms. This means OpenVPN has been widely implemented across the industry, even if it’s slightly dated compared to modern releases. IPVanish advises all users to pick this protocol whenever possible.

OpenVPN uses OpenSSL for encryption and authentication, with a choice between UDP (User Datagram Protocol) or TCP (Transmission Control Protocol) for transmitting the data to and from your device.

The main difference between the two is that UDP is faster. Although, it’s more likely to encounter errors due to lost data packets. This means that TCP is often preferred unless it’s for an intensive task like online gaming.

pros

  • Open-source software licensed under GNU GPL
  • Widely accepted as one of the most secure protocol options while using TCP
  • Free to download and use so long as you can configure it or your VPN offers it outright

cons

  • Slightly dated compared to modern releases
  • Not the fastest protocol, but it’s available with most commercial VPN apps 
  • Requires third-party software and certificate files to be installed

L2TP

Layer 2 Tunneling Protocol (L2TP) is used to support VPN networks, although it provides no encryption. Due to the lack of privacy, it’s often used in conjunction with a protocol like IPsec, which is discussed below. This combination is called L2TP/IPsec.

L2TP/IPsec works by authenticating the data transmitted twice. This has an impact on connection speeds, but it should provide some of the best security measures of the protocols to make the list. It’s the successor to PPTP, which we cover below.

There are no known weaknesses with L2TP. However, the likes of Edward Snowden and EFF founder John Gilmore have alluded to potential flaws as the protocol may have been weakened by the NSA in the past. (To this end, Hide.me advises against using the protocol, and bVPN ended support in August 2021.)

pros

  • Widely compatible with the majority of platforms and operating systems 
  • Uses AES-256 bit encryption
  • L2TP/IPsec is a solid combination for encryption and authentication

cons

  • L2TP/IPsec uses UDP port 500 which is easily detected and blocked by firewalls
  • L2TP on its own is not very useful
  • Not as widely available across premium VPN providers

IPsec

As mentioned above, IPsec is a protocol that is used to secure data sent over public networks. It works by encrypting IP packets and authenticating the data that is transmitted. “IP” stands for “Internet Protocol” and “sec” for “secure.”

IPsec uses UDP because this allows IPsec packets to get past firewalls. It can work in either Transport mode or Tunnel mode, with the latter the default option. (Tunnel mode encrypts the entire data packet, while Transport mode is ideal for secure communications.)

As well as L2TP, IPsec can be paired with other protocols such as IKEv2, which we cover below. It can also be used as a standalone VPN solution.

pros

  • Solid authentication
  • Known for its robust network security
  • Available via transport mode or tunnel mode

cons

  • IPsec will have an impact on speeds and performance while in use
  • There are potential restrictions due to firewall issues
  • IPSec has an inconsistent standard for its own compatibility

SSL/TLS

Secure Socket Layer (SSL) is used to encrypt data packets, ensuring a secure link within the browser. SSL certificates are also helpful, as you’ll definitely be connecting to the correct server. 

SSL was released in 1994 by Netscape Communications Corporation in an effort to secure their web sessions. The Transport Layer Security protocol (TLS) is the successor to SSL, and is more commonly used by modern VPN services. TLS was introduced in 1999 to mitigate serious security flaws found with SSL.

Due to their similarity, SSL and TLS are often used interchangeably, even if it’s usually describing the latter. (TLS v1.0 release began development as SSL v3.1, which is why you’ll find that the two terms are often mixed up.)

pros

  • An SSL VPN provides end-to-end encryption between the VPN client and its servers
  • SSL used in online payment authentication due to its high security
  • Google algorithm prefers SSL over other authentication methods to rank pages

cons

  • An SSL VPN will actually use TLS due to serious security flaws found within the protocol
  • OpenVPN is an example of an SSL VPN, so it’s not compatible with IPsec or L2TP
  • Not as commonly used by VPN providers as OpenVPN or IKEv2

SSTP

Originally developed by Microsoft as a proprietary option, SSTP (Secure Socket Tunneling Protocol) uses an SSL/TLS channel. It was introduced to coincide with the release of Windows Vista. 

It has the ability to bypass firewalls easily, as well as accessing blocked content. However, SSTP was designed with Windows devices in mind, so it’s not the best option in terms of compatibility. 

pros

  • Seen as a more secure alternative to the likes of PPTP or L2TP/IPsec for Windows
  • SSTP may offer similar speeds and security to OpenVPN
  • Helps bypass firewalls better than other protocols

cons

  • Being solely owned by Microsoft isn’t ideal from a privacy perspective 
  • Can have an impact in terms of connection speeds while the VPN is active 
  • Not ideal for non-Windows software like Linux or macOS

PPTP

The older brother of SSTP, point-to-point protocol (PPTP) has been around since the Windows ‘95 era. As you may have guessed, it’s now obsolete. We wouldn’t recommend its usage for anything other than accessing faster speeds on ancient hardware. Or, of course, use it when it’s your only protocol option. 

It’s susceptible to multiple known vulnerabilities due to weak encryption.

pros

  • It’s a fast protocol option, especially on older devices
  • Native built-in support for many popular operating systems including Windows, Android, MacOS and iOS
  • Fairly easy to set up on devices if not offered by VPN provider outright

cons

  • Obsolete, because of various security vulnerabilities
  • Many issues stem from Challenge/Response Authentication Protocol (CHAP), or inadequate hashing algorithms
  • Less reliable compared to SSTP since it lacks data origin verification and data integrity process

WireGuard 

The much-lauded WireGuard is a newer protocol. It aims to simplify the encryption process thanks to less code and a cleaner design. This means that it’s a viable option in terms of security and can be audited easily. It was created by Edge Security cryptographer, Jason A. Donenfeld. WireGuard’s first stable release, or version 1.0.0., was on March 29, 2020.

WireGuard is exceptionally fast and is often used when we conduct speed tests for review purposes. However, it is still seen as an experimental protocol. This means it hasn’t been widely implemented by providers as of yet.

For a more extensive look at WireGuard, check out our direct comparison with OpenVPN

pros

  • One of the fastest protocols overall, making it great for streaming and torrenting
  • Roughly 4,000 lines of code vs OpenVPN’s 70,000 lines of code
  • Strong encryption, with no known security flaws 

cons

  • Only a handful of premium VPNs offer it as its so new
  • Security concerns as it hasn’t been vetted as long as protocols like OpenVPN
  • It requires separate servers, distributions, and key management, which is another hurdle to VPN providers supporting this protocol

IKEv2

Often used with IPsec, Internet Key Exchange version 2 (IKEv2) is a popular solution that offers a good balance of fast connection speeds and strong encryption. It was originally released by Microsoft, who worked with Cisco to release the upgrade to the Internet Key Exchange in 2005.  

IKEv2/IPSec uses a Diffie–Hellman key exchange algorithm and supports AES 256-bit encryption. There are no known security flaws, and it’s a stable option that is quicker than the majority of the competition. ProtonVPN uses an open-source version of IKEv2/IPsec for its service.

pros

  • IKEv2 is one of the faster protocols available considering the level of encryption 
  • Typically used with IPsec, another reputable protocol for privacy
  • Good stability, and deployed by many commercial providers 

cons

  • Easier to block with the use of firewalls due to the use of UDP port 500 
  • Offers limited support for devices outside the Windows and Apple ecosystems
  • Some open-source versions exist, but it’s primarily a close-sourced development of Microsoft and Cisco

Additional Proprietary Protocols

As the name suggests, proprietary protocols are typically developed and used by a single company, or in some cases are licensed out for further use. They claim that it gives them an edge over the competition listed above, as it will have been specifically designed to work with their network. Notable ones include Lightway, Catapult Hydra, and NordLynx. Let’s take a quick look at each.

Lightway is a very new VPN protocol released by ExpressVPN in the summer of 2021. They note that, “nine out of ten beta users reported that Lightway got them connected to the VPN faster than before.”

Another example would be Catapult Hydra by Hotspot Shield and its parent company, AnchorFree. According to them, “AnchorFree used to use standard IPSec and OpenVPN protocols to power Hotspot Shield but found major performance and latency challenges with it, therefore we created our own proprietary Catapult Hydra to address the issues of VPN latency.”

There’s also NordVPN’s repackaged version of WireGuard that they branded as NordLynx. They claim that with it, users can “experience WireGuard’s speed benefits without compromising your privacy.”

pros

  • Proprietary protocols tend to be responsive as they were designed solely for the VPN service
  • They’re still a worthy additional feature and are always worth testing out
  • Speciality protocols can be very attuned to specific use cases

cons

  • Many are still in beta mode, and may not be as stable as other options
  • Expect VPN services to embellish the results of their tests ever so slightly. (It’s unlikely that speeds will be many times faster than using an OpenVPN/WireGuard/IKEv2 solution.)
  • Rarely available outside the VPN service which it was created for and by

Best Protocol for Speed

graphic of computer speed test

Speed is key to accessing content without lag or buffering, and it’s one of the most important aspects for the average VPN user. To this end, it’s hard to look past proprietary options that have been custom-built for the job at hand. Otherwise, we’d advise checking out WireGuard, especially as we use it for the purpose of speed testing whenever the protocol is available. 

IKEv2 should be a little faster than OpenVPN, while the barebones nature of PPTP also makes it speedy. We’re big supporters of WireGuard, but we do recognize that it’s still in development. 

Overall, The best protocol for speed depends on the task at hand, as well as your typical internet speeds, and the provider selected. Surfshark is our top pick for streaming thanks to P2P support, while CyberGhost has a massive high-speed server network. 

Best Protocol for Privacy   

graphic of lock and passcode

Privacy is often disregarded in favor of features like access to streaming services, but it’s becoming ever more important as multiple companies compete for our personal data. 

The best protocol for privacy has to be OpenVPN at this moment in time, as it’s the industry standard for a reason. IKEv2 is a great pick for the strongest encryption and security, although it’s worth mentioning that WireGuard has no known major vulnerabilities.

We’d couple the use of OpenVPN/NordLynx with NordVPN for the best results in terms of security and privacy. IPVanish also has robust security features. Further options include ProtonVPN, while Mullvad is an audited privacy-focused service that uses OpenVPN and WireGuard for tunneling. 

Best Protocol for Compatibility   

graphic of devices

What if you plan to use a protocol across different devices, and a range of operating systems? The ideal protocol for compatibility has to be OpenVPN, given it’s a solid option that can be used across a wide range of apps and devices. In comparison, WireGuard is great, but it’s not readily available with many VPN services. 

However, the “best protocol” might be dependent on the VPN provider you’ve selected, as they are likely to have various options that differ depending on the platform. 

For example, NordVPN offers OpenVPN, IKEv2/IPsec, and WireGuard via NordLynx, with their proprietary protocol used as the recommended setting. 

IPVanish has the following protocols available including IKEv2, OpenVPN, L2TP, IPsec, and PPTP. (OpenVPN works with every operating system, whereas WireGuard is only available for Windows, iOS, macOS, Android, and Fire TV.) To access Chrome with IPVanish, you’ll have to use OpenVPN or L2TP. 

Best Protocol for Content   

graphic of IoT devices

Isn’t the best VPN protocol for streaming the same as the best VPN protocol for speed? Not necessarily, as it’ll be more dependent on whether the VPN is able to access content like US Netflix in the first place. They will also need local servers in the location you would like to connect to, and a constant list of new IPs.

Protocol selection will have an impact on speeds, and the same is true for streaming/torrenting. Check out our guides for the top VPNs for torrenting and the top service for streaming

Final Thoughts

As you can see from the examples above, the protocol selected will have a major impact on a VPN’s performance and security. Some protocols are now obsolete, offering poor levels of encryption that are easily cracked, while others are still in the development stage but make for viable options in the long term. 

Ideally, a VPN provider will have a large selection of protocols to select from within its app, giving the user the opportunity to decide which is best for any given scenario. At the very least, OpenVPN and IKEv2/IPsec should be included as standards.

Explore the best VPNs in 2021 to see which one offers the right protocols for your use case.