Configuring a VPN tunnel from Fritzbox 7390 to Juniper SSG5

Many people seem to be following the guides on how to set up infrastructure VPN using Fritzbox and still can't get it to connect. If the Fritzbox logs anything at all about the problem it's just a cryptic error number. You can look the numbers up easily enough, but the short phrase "timeout" or "bad hash" doesn't do much to narrow down the problem.

The real cause of mysterious 0x2020 and 0x2027 errors and connection failure

If you're prepared to struggle with the numerous German language postings on the 7390 you'll find that while many posters insist that the recipe for VPN success is simple, others can't seem to make it work no matter what they try.

When you copy all the instructions verbatim, then try everything else as well, and all you get is <nothing> in the Fritzbox log, or a bunch of cryptic 0x2020 or 0x2027 errors, the problem is almost certainly related to MTU. While this ought to be fairly easy to work out, I haven't found a single post on the topic that mentions MTU even once. If you have tried everything with your Friztbox, then MTU is probably the problem - it seems sensitive to MTU when other boxes get by just fine with a default value.

The Juniper SSG5 allows you to dump the entire packet trace of an attempted IPsec negotiation, revealing that the Fritzbox sends some questionable messages regardless of configuration, and these will result in 0x2027 errors, but they don't stop it eventually connecting. However, the 0x2020 errors indicate a fatal problem, almost certainly related to MTU settings.

But the Fritzbox won't let me set the MTU!

The interface and options on the manufacturer's standard shipped interface to the 7390 are limited indeed. It wasn't until a recent update that we got the ability to set-up a VPN without recourse to a laughably poor tool from AVM that only ran on windows, despite the fact that it could have been knocked up in a few dozen lines of python or ruby.

AVM themselves say:

The FRITZ!Box does not allow you to adjust the MTU size manually. This is also not necessary because the FRITZ!Box supports the MSS clamping procedure (Maximum Segment Size) that automatically adjusts the size of data packets being sent to a connected network with a smaller MTU. Hence there will not be any performance losses due to fragmented or discarded data packets.

This is all very well, but it's quite clear that the Fritzbox cannot reliably detect LAN routes and adjust MTU accordingly so that tunnel packets don't become mangled.

Short of opening up your 7390 with modified firmware using Freetz you cannot get around a genuine MTU issue on the Fritzbox, and if the other end of your tunnel is another Fritzbox, then you are genuinely out of luck. However, most people with VPN problems are trying to connect to somewhat more transparent hardware, such as Netgear ProSafe, Draytek Vigor or perhaps a Juniper SSG5 - just about every other manufacturer out there allows MTU adjustment, even cheap TP-Link boxes allow it. The inability to set MTU for VPN tunnels is a serious oversight.

Setting the MTU on a Billion, Belkin, Netgear or Draytek is simplicity itself, but configuring VPN tunnels on the Juniper SSG involves an understanding of a wide range of options and complex choices, so many may be in need of some help in that regard.

The Fritzbox Side

First create your Fritzbox configuration file

Don't bother with the utility program. All you need is a simple text file in ANSI/OEM format. I'm actually somewhat uncertain exactly what encoding is required, though reports suggest that UTF8 does not play well with the parser. In any event, you won't need to input any strange character codes unless you are trying to use a FQDN with fancy characters in it and I'm not even going to talk about using a domain name or dynamic DNS here.

It's said that you should edit your VPN configuration file in DOS mode (with CR+LF line endings). Whether this is absolutely necessary I'm not sure, but DOS mode files do work, so you should probably stick to that, and if using a Linux based editor, it would be best to set it to use CR+LF endings. The Fritzbox saves out its main configuration file backups in Linux format (LF line-endings), but AVM's VPN utility saves out DOS format files and I've never tried loading Linux line endings into the VPN setup.

Your configuration file should look something like the following

vpncfg {
        connections {
                enabled = yes;
                editable = no;
                conn_type = conntype_lan;
                name = "ddd.ddd.ddd.ddd";
                boxuser_id = 0;
                always_renew = no;
                reject_not_encrypted = no;
                dont_filter_netbios = yes;
                localip = 0.0.0.0;
                local_virtualip = 0.0.0.0;
                remoteip = ddd.ddd.ddd.ddd;
                remote_virtualip = 0.0.0.0;
                keepalive_ip = rrr.rrr.rrr.rrr;
                localid {
                        ipaddr = fff.fff.fff.fff;
                }
                remoteid {
                        ipaddr = ddd.ddd.ddd.ddd;
                }
                mode = phase1_mode_aggressive;
                phase1ss = "all/all/all";
                keytype = connkeytype_pre_shared;
                key = "yoursharedsecret";
                cert_do_server_auth = no;
                use_nat_t = no;
                use_xauth = no;
                use_cfgmode = no;
                phase2localid {
                        ipnet {
                                ipaddr = fff.fff.fff.0;
                                mask = 255.255.255.0;
                        }
                }
                phase2remoteid {
                        ipnet {
                                ipaddr = rrr.rrr.rrr.0;
                                mask = 255.255.255.0;
                        }
                }
                phase2ss = "esp-all-all/ah-none/comp-all/pfs";
                accesslist = "permit ip any rrr.rrr.rrr.0 255.255.255.0";
        }
        ike_forward_rules = "udp 0.0.0.0:500 0.0.0.0:500", 
                            "udp 0.0.0.0:4500 0.0.0.0:4500";
}


// EOF

Unfortunately, AVM don't make much effort to document their configuration file format, options or behaviour, instead expecting users to rely on a fairly inadequate Windows utility and highly inadequate built-in configuration tools. While the Fritzbox has a great feature-set on paper, the restrictive firmware often lets users down, and in the case of VPN it can cost a lot of time and effort to get a working configuration. Using information from the German forums and experimentation, I'm taking a best guess at how the options function.

  • fff.fff.fff.fff is the local address (range) on the Fritzbox. For sake of our example we'll assume it's 192.168.10.0
  • ddd.ddd.ddd.ddd is the external (fixed) IP of the remote endpoint.
  • fff.fff.fff.fff is the external (fixed) IP of your Fritzbox endpoint.
  • rrr.rrr.rrr.rrr is the local gateway IP at the remote end, e.g. 10.1.1.1 or 192.168.1.1 - this presumes that you have some NAT going on at the remote end, but you probably do. Let's take for granted you know how to set up NAT on your SSG.
  • rrr.rrr.rrr.0 is the generic address space of the local network, e.g. 10.1.1.0 or 192.168.1.0. If your net-mask is actually 255.255.0.0, then you would use 10.1.0.0 etc.
  • We're assuming /24 local networks at both ends here, so the 255.255.255.0 values are net-masks.

The pre-shared key must match the value you put in the other end of the tunnel exactly. Make sure you don't have any invisible characters hiding on the end and it isn't longer than 49 characters (or shorter than 8). The Fritzbox will allow a short string here, but other boxes may not, or may even impose a fixed length (which substantially reduces the key-space, but does prevent easily guessable keys).

Do not confuse the "reject_not_encrypted = no" setting with somehow enabling unencrypted transmission, that's not what it does - if set to 'yes' it would force the local network specified in the tunnel definition to use only the tunnel route - which would force all external traffic through the tunnel. If that's what you want to happen, set it to yes.

You may have noticed the line "mode = phase1_mode_aggressive;"... Some sources will suggest that you use main mode for a pair of static IPs. This is good advice in general, but I have not had any success at all the 7390 negotiating a link with the SSG in main mode, despite having two fixed IPs. You can try setting "mode = phase1_mode_idp;" and setting your other end accordingly. If you're using a different version of ScreenOS to me, or have a different box entirely, perhaps it will work.

You may also wonder that I've chosen to turn off NAT traversal. While this option does work, and you can use it if you really need it, try without it first. Sometimes turning this off can get rid of annoying timeouts during negotiation, even if the negotiation is eventually succeeding.

"editable = no" prevents editing this configuration in the new Fritzbox GUI. It would probably only mess it up as it's only intended to handle a very narrow range of configurations.

The values chosen for the phase1 and phase2 configurations are almost the vanilla values set by the AVM program. You can try playing about with these, but I wouldn't bother. The Freetz people are saying that the list of supported strings has been substantially reduced under 6.20+ versions of the firmware. A value you might actually want to try out is "dh5/aes/sha" as this is a reasonable combination of DH group and AES, but there doesn't appear to be a phase2 equivalent, and all phase 2 configurations seem to be group 1 or 2.

Navigate the Fritz web interface to Internet >> Permit Access >> VPN and hit the "Add VPN Connection" button. If you have a choice, choose to upload an existing configuration file (Import a VPN configuration from an existing VPN settings file). Hit "Choose File" then locate the file on your local machine and OK it. Wait, wait, wait. After a while the file name will pop up next to the Choose File button. Once it shows up you can hit OK to perform the upload. The Fritzbox will show you a blue busy bar for a while ... or if you messed up the configuration file completely you'll see a red bar and a failure message.

On the VPN page ensure that the tunnel is enabled and then move on to the other end...

Configuring a Fritz 7390 compatible tunnel on the Juniper SSG5

I'm going to discuss how to create a route based tunnel here. Some examples show policy based tunnels, but you must have a concrete tunnel interface to control MTU, and any policy based tunnel that uses a tunnel interface binding is complex; you probably don't need that sort of set-up. On the other hand, a policy-based tunnel is very easy to set up if you want the most basic kind of tunnel using a tunnel action, but it won't cut it in this case.

In the examples below, ethernet0/0 (trust-vr) is connected to the internet - and it's the physical interface I am going to run the tunnel through. If you have configured your SSG differently, then substitute your local value for ethernet0/0 (trust-vr) accordingly.

Create the Tunnel Interface

This is done from Interfaces >> New [Tunnel IF].

Create the Tunnel Interface on the SSG

The example above uses tunnel.3, but yours is unlikely to be the same. The interface will offer the first unused tunnel. Just make sure you refer to the same tunnel you create here, later, when binding the VPN.

You need to create an explicit tunnel interface so you can set the MTU on it, and you can see that I've set it to 1300 here, which is about as small as you would ever need to go. Depending on the nature of your problem, a larger setting might work. If 1500 (or 0) works then you don't have an MTU problem :)

It's a useful short-cut to put the interface in the untrust zone. Complex scenarios using a custom zone also work, but if you're happy doing that you don't need this guide. Do not put the interface in the trust zone; this doesn't work well at all and may result in a one way tunnel regardless of the policies you set, or not work at all, depending on how your trust zone and other (real) interfaces are set up - due to a requirement for non-obvious routes and policies.

Create the Phase 1 Proposal Type

The SSG doesn't have the Phase 1 AES256 + SHA-1 pre-shared option that the Fritzbox offers first, so we need to create it.

Create the phase 1 proposal on the SSG

You'll need to select this proposal in the gateway advanced settings later.

Create the Gateway

The basic gateway settings are something like this; these are the defaults in fact. All you have to do is fill in the external IP address of the Fritzbox.

Create the gateway on the SSG

Modify the Gateway Advanced Settings

Unlike the basic settings, there is a bit more to change here. Fill out the pre-shared key to whatever value you're using and set the phase 1 proposal to use the value you created previously.

Don't tick NAT Traversal until you've determined that you really need it. If you do set it, it must match the value in the Fritzbox configuration file.

Note that you don't have to set this just because you have NAT on your local network; it's nothing to do with that. This is mainly useful to work around NATs you can't see, such as stealth NAT or blackhole routing by ISPs or co-location providers. Setting this value restricts UDP communications to a single port. In theory it should be more dependable, but it also adds some overhead. I've found it isn't always necessary, even if there is a NAT in play. In practice, such NATs may forward the UDP traffic you need anyway; your mileage may vary.

Set the gateway advanced settings on the SSG

The pre-shared key value can be anything as long as it matches the value you put in your Fritzbox configuration file, but a random string of characters would probably be best. 49 characters is usually the upper limit in length.

Create the VPN

The menu option is actually called AutoKey IKE because it's distinct from manual keying.

Create the AutoKey IKE VPN on the SSG

All you have to do here is pick the gateway you created previously and then move on to the 'advanced' settings.

VPN Advanced Settings

After picking the gateway, click the 'Advanced' button to set up the important stuff.

Continue to create the AutoKey IKE VPN on the SSG

Set up the phase 2 proposal. You only need one value, this will be the first thing the Fritzbox suggests.

Set up the address mapping. As with the example Fritzbox settings, these may vary for your local net. Here 10.1.1.0/24 is the local address range on the SSG bgroup1 and 192.168.10.1/24 is the address range on the Fritzbox.

Select the tunnel binding and select the tunnel interface you made at the start. Make sure this is the correct tunnel interface. The example uses tunnel.3 but your tunnel may have a different number - make sure it's the one you created earlier.

Turn on VPN Monitor, Optimized and Rekey. If you try to use monitor without optimized, it probably won't work. Monitor causes the SSG to use ICMP packets to monitor the connection status to the other end of the tunnel.

Choose a 'Destination IP' for VPN Monitor that will always be there and able to respond to ping, such as the Fritzbox itself. The IP is a local IP, so 192.168.10.1 would be the obvious value in this example. If you don't fill it in the SSG will guess.

Rekey causes periodic reconnection, regardless of tunnel use. If you don't set it, your tunnel will drop when there is no traffic over it. This can be a problem because it then has to renegotiate to bring the tunnel up on demand, which takes a long time and can cause timeouts, so you most likely want it set.

Create the tunnel Route

Because this is a route-based tunnel, it won't work without a route to direct traffic into it.

We will be using a regular destination based route. The destination in this case is the local IP range on the Fritzbox, which is 192.168.10.0/24 in our example (fff.fff.fff.0 from our Fritzbox configuration file). We want to route this destination through our tunnel (in the example, tunnel.3).

Set up the tunnel route on the SSG

Select Network >> Routing >> Destination and fill out the destination address range, then select 'Gateway' and pick the required tunnel interface; yours may be a different tunnel number, make sure it's the one you bound to in the previous step. You should be using the same tunnel interface you created in the very first step.

You don't need a reverse route; you should already have a suitable route. If you could already route to your local interface (in the case of the example bgroup1) then your tunnel packets will also be routed there. As long as you put this route in the correct virtual router it will work. I've put everything in trust-vr for this example, as that is the most obvious use case.

Create the tunnel Policy

This isn't a policy based tunnel, so a policy may not be needed. If you have the default trust -> untrust ANY policy, then that will cover it for outgoing packets. You do not need a tunnel policy, and you would not be allowed to add one anyway because the tunnel is already bound.

You probably need a policy for incoming packets, such as the one shown below.

Set up the tunnel route on the SSG

I would normally use a pre-defined address range set up in Policy Elements >> Addresses here instead of simply putting in the address range directly, but it's clearer as an example this way (and avoids some steps that are really nothing to do with VPN setup).

Note that there is no tunnel setting here, it should be a regular policy with PERMIT as its action.

Supported Fritzbox Configurations

Phase 1

According to the Freetz postings, the following are the only supported combinations for phase1ss

  • dh5/aes/sha
  • dh14/aes/sha
  • dh15/aes/sha
  • def/all/all
  • alt/all/all
  • all/all/all
  • LT8h/all/all/all

Phase 2

And these are the only supported phase2ss combinations.

"pfs" relates to "Perfect Forward Security", which reduces your vulnerability to initial key compromise attacks. The SSG5 supports this, so you can turn it on - this is done simply by not selecting a Phase 2 proposal option with nopfs - if it doesn't say nopfs, pfs is assumed.

The cryptic LT8h appears to refer to live-time-8 hours. People have reported issues with this so I didn't try it.

  • esp-3des-sha/ah-no/comp-no/pfs
  • esp-3des-sha/ah-no/comp-no/no-pfs
  • esp-aes256-3des-sha/ah-no/comp-lzs-no/pfs
  • esp-aes-sha/ah-all/comp-lzjh-no/pfs
  • esp-all-all/ah-all/comp-all/pfs
  • esp-all-all/ah-all/comp-all/no-pfs
  • esp-all-all/ah-none/comp-all/pfs
  • esp-all-all/ah-none/comp-all/no-pfs
  • LT8h/esp-all-all/ah-none/comp-all/pfs
  • LT8h/esp-all-all/ah-none/comp-all/no-pfs