As I think about what I want out of this project and some of the designs I could employ, it occurs to me that I could easily be tempted to turn my entire home into one giant experiment. Even if I didn't have a wife and two grown children living with me right now, I depend on my Internet connectivity myself for various work throughout the day. Having to scramble to patch together Internet because I was right in the middle of reconfiguring the lab and I have to jump on a client call would be bad...and then there's the family that would scream bloody murder if the Internet were going down 5 times a day. Granted, I could always blame the outages on Comcast, but that's besides the point. The realization that I have "users" on the "enterprise" network that demand a certain level of stability is good, because it starts to drive some thinking about what the key principles in our design are.
This is a process I follow before I start on any design for a client. First I figure out what the main goals are. Is it cost reduction? Is it bandwidth/speed improvement? Is it improved resilience? Is it agility? Then we figure out what the constraints are; money, downtime restrictions, time to implement, personal/organizational opinions/religion, etc.
Objectives/Design Principles
1) flexible design/easily reconfigurable to maximize learning opportunities
2) not mission critical - can be rebooted, or turned off for long periods of time without affecting other users on the "enterprise" or consumer network (e.g. family)
3) can be accessed by general users on the Internet on an ad-hoc basis to support demonstrations or other needs
First Draft of Design
Ideally, I could run my lab network in parallel with my consumer-grade WiFi Mesh Product (Google WiFi). That would involve having a switch domain plugged into the cable modem that simultaneously connect to my Mesh Router as well as a secondary router that sits on top of my lab servers. It would also require that Comcast assign me more than one dynamic IPv4 address, which I didn't think was going to happen, but it was worth a test to confirm and my suspicion's check out...only one IPv4 assignment...and IPv6 seemed to fail all together during this process as well (by fail, I mean I wasn't getting any IPv6 addresses on any devices).
So I looked into "upgrading" my residential service to business class with Comcast. Right now I get a triple play (Phone/TV/Internet) 600Mb service for around $200/month. My options on business class are 300Mb for $160/month or a full Gig for $500/month, and each static IPv4 address is another $25/month. I spoke with a rep on the phone that explained to me that due to regulations, they can't offer me a residential TV service with a business class Internet, so I would have to cancel the Internet on my residential service and buy just TV. We're not ready to "cut the cord" quite yet (well, my wife isn't) so all in I'd be looking at a 50% increase in cost to get static IP's only get half the bandwidth. And oh, by the way, my upload of speed of 35Mb wouldn't change either. I asked the rep about static IPv6 and she asked me what that was (EPIC FAIL), but googling myself I found this informative blog post that explains that business customers are assigned a /56, but due to the way cable modems work today (I have a Netgear CM1000), the addresses are pulled from Comcast a /64 at a time just as they are on the residential service (more on this in a separate post). This is just all kinds of 'not ideal', so I'm passing on the business class upgrade.
Second draft of ideal design (scrapped)
If I put a router on top of the design and drop everything else, including the mesh wifi under it, I could accomplish what I want, but then I have part of my experiment putting the stability of the whole network at risk (this router would run in an virtual machine, at least initially). I envision these boxes getting rebooted fairly regularly, even re-installed from scratch on occasion, so this design just isn't going to work. But it could, with some tweaks...
End State Design
In order to solve for the reliability issue, I think I'll want a reliable device with good routing features that I won't mess with too often so that I can have multiple routed subnets behind the box while it does NAT. The Google WiFi router doesn't fit this bill, as it doesn't have the key features I'm looking for. At a minimum it would have the ability to add static routes so that I can subnet, but ideally I'll want something that can manage multiple Internet connections as well. From a hardware perspective, I'll probably want to get a basic network appliance...something with a Xeon processor in it so there's native support for virtualization (Lanner has several that fit the bill, as does Dell). I could probably start with VYos, an open source router built on top of Linux, and maybe move to something SD-WAN-ish later on if I can find a design that will work. I'm not ready to drop $1,000 on one of these just now, so this is more of an end-state design:
I'm thinking my end-state design will look something like this:
Until I get to that end-state, I'm going to have to live with something pretty simple and straight forward.
All of my devices will have to hang off a single IPv4 subnet for now since the Google WiFi router doesn't support static routes, and any services I want to expose to the Internet will have to be port-mapped for IPv4 on the Google WiFi router. For IPv6, I should be able to expose the addresses directly, but more on that the follow-up post I promised on IPv6.
And last but not least, my logical design will ultimately look something like this.
Siegel Group does all of its consulting through Yates Ltd. Yates Ltd is a boutique consulting firm specializing in cost optimization and network transformation. If you could use some outside help in your business, reach out to me to see how we can help.