How to get Mac OSX VPN to work with BT Home Hub 2.0

After much gnashing of teeth and utter confusion, I have finally found the magic sauce that makes VPNs work through ASDL modem routers, specifically the hard-to-configure but very popular BT Home Hub 2.0. These instructions apply to Apple Mac OSX Lion 10.7, so  they won't work on Windows but they might provide you with some clues.

First, here's what I'm using:

• A BT home hub 2.0 connected directly to my phone line
• An Apple Time Capsule (3rd Gen), connected to the BT hub. Nothing else is connected to it so it should have no effect on this setup
• Comtread powerline 9020 networking doo-dahs. They send Ethernet over powerlines
• A Mac Mini connected to the powerline ethernet
• A whole bunch of other stuff that's too boring to mention and really didn't effect anything
• An iPhone to test the VPN via 3G
• a DYNDNS account, to give it a domain name on the Internet

The first thing to do is to install Lion, and the Lion Server tools from the Mac App Store. Once they are installed, set up the VPN configuration - all it needs is a shared secret phrase and a range of IP addresses. These should be addresses within the same subnet as the ethernet connection.

Second thing I did was to give the VPN server a static IP address on my network; change the Mac to use DHCP with manual address, and give it an address outside the DHCP range on the home hub but on the same subnet. You could use something like 192.168.1.10 if you haven't changed any of the defaults. 

Thirdly, completely ignore the nonsense in the Home Hub about assigned applications - it doesn't seem to work. Download the application from http://www.codingmonkeys.de/portmap/index.html and run it on the VPN Server. I know the piratey theme to the site doesn't inspire confidence, but the software seems to work quite well. So well in fact that it's invaluable when diagnosing odd behaviour with uPnP and dynamic router configuration. I couldn't have made this work without this software - it gave me the clue that I needed to turn off 'Back to my Mac' for this to work.

When you launch the Port Map application, it should find your Home Hub quite quickly and tell you it's external IP address. Click the 'all UPnP mappings button'. You might have some already - I did. You're looking for ports 500, 1701, 1723 and 4500. According to some sources, technically only 500 and 4500 are needed but I added them all, mostly because I'd spent so long trying to get this to work. If you have these in the list then they are mapped to something else and you're going to be having failures if other devices steal these ports back. Also, the remove feature doesn't work with the Home Hub (I think it's meant to for other brands). I hit reset on the hub and they all vanished, UPnP port forwarding is a temporary and not permanent condition, although it seems to be persistent for the hub until reset is hit. 

Add forwarding for port 500, 1701, 1723 and 4500 for both TCP and UDP. Incidentally I got these numbers from an Apple support document detailling common port numbers; all of these are listed as VPN ports although 1723 is PPTP which I believe is no longer used. Once you have the four ports 'switch' them to the on position. You should have the right mappings in there now. The public port and local port MUST be the same for this to work. If not, you'll need to reset the hub and try again.

Note that MobileMe 'Back to my Mac' steals the ports you need. Turn it off if you want a VPN. If you want to access files at home just dial in with the VPN then use screen sharing, ssh, file sharing etc etc.

All that is left is to put the details into a laptop or a phone with VPN capacity, it's now set up as L2TP. Enter your shared secret, username, and password. Of course, you will already have entered your DynDns details into the home hub so that you can use that domain as the target for the VPN connection. If all is well, you should now be able to connect and access all your home network safely and securely. I've also enabled 'Send all traffic' on my iPhone so that I have some protection against some of the nastier things out there when using public data connections such as FireSheep.

Liked this article, got a problem, or would like to leave feedback. Leave a comment... Thanks!

Update on Saturday 30 July 2011 at 19:19 by Antony

The list of common ports used by OSX is here: http://support.apple.com/kb/ts1629

Comments

[Duncan]

Hello! Thanks very much for the detail. I've found it very useful indeed.

I have a (quick?) question - I have a similar set up to you - Internet -> bthomehub 3.0 -> Time Capsule -> all my clients (imacs, macbooks, iphones etc).

I want to stick a mac mini running Lion Server onto the network after the Time Capsule. But I want to be able to configure portforwarding and DHCP services on the TimeCapsule from the server (which I believe the server can allow you to do).

(at the moment my homehub provides the DHCP service and my TimeCapsule bridges to the rest of my network),. Would it be better to place the TimeCapsule in the DMZ of the homehub and disable the firewall of the homehub as well. Does that then 'open up' the internet to my TimeCapsule and thereby give it control of which ports are open?

If that won't work, I assume it's back to manually managing the portforwarding on the router and leave my TimeCapsule as a bridge?

Thanks for any info

[Antony]

Hi Duncan,

Thanks for the feedback, I'm glad it was useful. It does sound like a similar setup, but for one major difference - BT homehub 2.0 vs bt home hub 3.0 - I have no idea what they changed, or even who manufactures these router modems. Also, my timecapsule is used as the wifi point for my laptop and iphone, but not for the mac mini - it's wired with Ethernet into the bt hub using powerline connectors. The timecapsule is set up as a bridge.

I use the wifi on the timecapsule as it's far more reliable than the hub. It keeps dropping signal for me.

The PortMap tool is key, it can read all the UPNP settings from the Home Hub 2. Try it on the Mac Mini with your hub. One oddity is sometimes it comes up with 'router incompatible' then refuses - relaunching the tool fixes this.

I've not played with DMZ settings, quite frankly because I don't want my time capsule on the public internet. I'd stick with manually managing the ports, there's only four...

Truth is that this worked, then didn't work, then did work for a while. It's not reliable for me. There's something strange with the whole setup. As far as I can tell, somewhere it refuses to route to the Mac to make the connection. I think it's the home hub, but v2 is flaky at the best of times. I hope v3 is much better. You've inspired me to go fix it, I'll let you know if I make any progress...

Good Luck!
Antony