Background
I created my first homelab in 2012. At the time, the main goal was to learn Linux and get some experience in virtualizations. My main system was a 12.04 Ubuntu Server Edition. After getting comfortable with Linux, I started a basic webpage hosted in Apache where I would host various scripts, primarily written in bash, for my coworkers to use at work. This became wildly popular and I ended up transferring the data to a company intranet which was both more accessible and more secure.
The goals of my homelab have changed over time, but the push to evolve it further and utilize it to help me learn the latest technologies has always been the top priority. As I write this in early 2023, I am currently taking the next step in my homelab and converting everything to rack mounted systems. Before we get to the current hardware and services that I use, I'll talk briefly of past setups that were in the various apartments and houses that I lived in. I unfortunately don't have any pictures of my first setup, but I do have some from 2015 and beyond.
2015 to 2019
This was roughly my homelab from 2015-2019. It was in the closet of a guest room and consisted of an VMWare ESX host built into an old gaming PC with add-on 1 GB NIC's and locally attached storage. This was the only server aside from a backup appliance. After some time, I added two NAS devices, a 4 bay and an 8 bay, which hosted NFS and iSCSI storage for the ESX host. 10 Gbe NIC's were very expensive at this time so I decided to LAG various 1 GB NICs with a Cisco managed 24 port switch that I ended up getting later. Those were the main components alongside a wire rack to neaten things up a bit. It mostly stayed like this until 2018.
2019 to 2021
In 2019, I moved to a condo with less space. I brought the homelab but tried to consolidate it as much as possible. The biggest upgrade in this time period was a storage upgrade. I went from around 8 TB of worth of HDDs to a total to 60 TB. I did this the cheapest way possible, by waiting for a BestBuy sale of Easystore 12 TB external HDDs, and shucking them for the internal drives.
The lab was consolidated down a single, albeit somewhat powerful, Ubuntu server running ZFS in a RAID Z with 3 of the 5 shucked drives, as well as a backup appliance in a RAID 0 with the other 2 drives. I brought the Cisco switch from the last house and upgraded to a fairly large APC UPS unit to keep everything going during a power loss. I did eventually upgrade to an ASUS router which turned out to be a nightmare with it rebooting constantly for unknown reasons. For what it's worth, I have had great success with TP-Link products - switches, routers, POE injectors, APs, you name it. They are certainly a great value at their price point .
2021 to 2022
In 2021 I purchased a house. I put the homelab in my basement where the temperature would stay cool throughout the year, and I had CAT6 cables wired throughout the house. I brought the Cisco 24 port switch upstairs in my office and mounted various home automation hubs on the wall in the office closet (shown later on). This would be my workshop area where I would put my Prusa Mini 3D printer, Arduino and Raspberry Pi components, and some other miscellaneous items that I will go into in the next section.
The homelab in the basement was similar to the one I had previously with some small additions. I build a dedicated Blue Iris box which included an MSI 1660 Super GPU for AI processing via CodeProject.ai within the Blue Iris program. I threw in an M.2 NVMe SSD and an 8 TB WD Purple HDD and built it in a reasonably small tower knowing that I would be putting it in the little wire rack space I had left. I also added a PiKVM server with a 3D printed case on a Raspberry Pi 4 (2GB) attached to a KVM switch which allowed me to remotely access all of my servers as if a keyboard, mouse, and monitor were hooked up to all of them. While things like Teamviewer and RDP are great for remote access, the PiKVM allows me to get into the BIOS and even boot into ISO's that I keep stored on the PiKVM micro sd card. It is a very cool project and I highly recommend checking it out.
2023 - Present
Office Closet Workbench
While not technically part of my primary homelab build, I did want to share a few pictures of my workbench that I created in the closet of my office. If I have new hardware I went to set up but not rack in my basement yet, or components to solder for an Arduino project, I simply move my chair over to the other side of the room, turn on the lights (via a Home Assistant automation and Aqara button mounted underneath the table) and get going. I can also print components on my 3D printer. With the closet door closed, the noise is heavily reduced and isn't bothersome. I can of course keep an eye on the print even with the doors closed via the Reolink camera setup near it.
Some hardware included in this setup:
Reolink IP camera (w/ 3D printed mount)
Folding lamp (for extra lighting)
Z-Wave Smoke + CO detector (on ceiling, not shown)
26 Port Cisco Managed Switch (middle of rack, not shown)
Alinco DR-735T VHF/UHF Ham Radio (not shown)
20A Variable Power Supply (not shown)
550VA APC UPS (bottom of rack, not shown)
For those curious of what is mounted on the wall, from top left moving right in row-by-row format:
WiFi access point (replaced by ceiling mounted TP-Link AP (not shown))
Octoprint on Raspberry Pi
Amazon Echo Dot - solely used as a Spotify access point to play music around my house. Connected via aux cord to Pyle amplifier in the wire rack (not pictured).
Fire Tablet running Fully Kiosk and showing a Home Assistant sensors dashboard
Blink Camera server hub
NUT server on Raspberry Pi monitoring the APC UPS underneath it
SmartThings Hub v3 to connect various protocols to Home Assistant
Lutron Casseta Pro Hub to connect various smart switches to Home Assistant
Phillips Hue Hub
Homelab
This year will be the largest step I have taken thus far in upgrading my homelab. To start, I made the decision to move everything possible to rack mounted appliances. I like the idea of getting everything neat and organized and I'm going to put the rack under my staircase with a ventilation fan on a thermostat. That, and the ability to get rack mounted enterprise hardware such as the Dell PowerEdge R430 allows me a ton more functionality in my lab. While this server is a few generations old at this point, it is still a massive compute and memory gain, not to mention having improved hardware such as error-correcting memory, redundant power supplies, iDRAC, and various NICs at the ready. I will list some hardware that I am using below as well as their functions:
Hardware:
Dell PowerEdge R430 | This will be used as a VMWare ESX host - 2x E5-2690 v4 (28 core/56 thread) - 128 GB DDR4 RAM (+16 GB laying around) - 4 TB SSD - 3x 12 TB HDDs in a RAID 5 |
Dell PowerEdge 2420 Rack | 24U rack |
TP-Link 24 port managed switch | The Cisco switch is in my office, so I purchased a TP-Link switch for the rack |
CyberPower 1500VA/1000W UPS | This will be enough to power all of the gear. I also have a 650VA APC UPS connected to a different outlet on a separate circuit for redundancy and power distribution too |
Backup Appliance NAS | Syncs to the cloud. Can spin up virtualizations of backed up systems on the appliance. - i5-4400 - 16 GB DDR3 - 2x 12 TB HDDs in RAID 0 |
Blue Iris Server | Originally a tower, now in a Rosewill RSV-L4000U server chassis so it can live in the rack - i5-12400 - 16 GB DDR4 RAM - 256 GB NVMe SSD - 8 TB WD Purple HDD - 2 TB WD Black HDD - MSI 1660 Super GPU - Replaced fans with mostly Noctuas |
PiKVM + Ezcoo KVM switch | Connects up to 4 servers to the PiKVM |
TP-Link Router | This will get changed to pfSense |
ISP Provided Modem | Fiber internet |
AC Infinity 10" exhaust fan | 10" PWM fan, 3D printed rack mount for the included LCD thermostat display/controller. STL for this available on my Thingiverse. |
Monitor w/ MiniPC mounted on back | 3D Printed VESA mount I created to attach a MiniPC to the back of the monitor which is primarily used for server KVM functionality |
Software:
Previously, I had my primary Ubuntu server hosting everything, typically in Docker containers or VirtualBox VMs. With the new Dell R430 ESXi host, this functionality will be heavily split up. I am currently in the middle of setting this up. Below I will go over the functions the primary server, the backup server, the Blue Iris server, and miscellaneous Raspberry Pi's I have set up.
Raspberry Pis:
PiKVM - Remote KVM + BIOS management for various servers.
Octoprint - Remote 3D printer management, uploads, time lapses, etc.
NUT - Monitors my UPS's. If I lose power, I get notified.
Primary server
Virtual Machines:
Home Assistant (home automation, IoT)
Gaming Server (various)
PiHole (local DNS + ad block on LAN)
Windows 10 Sandbox
Ubuntu running containers (below), NAS shares, Discord bot, misc. services
Ubuntu running media services (mentioned below)
Docker Containers:
Obico (AI that senses 3D printing failures, AKA The Spaghetti Detective)
Portainer (Monitor/manage docker containers in a web interface)
Uptime Kuma (monitors + alerts on uptimes / down servers)
RTSP2MJPEG (lets my RTSP camera work with Obico + Octoprint)
Watchdog (restarts Docker containers on failures)
Jira (since moved to the cloud- my project to-do list, keeps me organized)
Miscellaneous Services:
Plex Media Server
Jellyfin (a Plex replacement that I am trying out)
Discord bot (I wrote in Node.js, performs various functions)
NAS sharing via various protocols for the LAN
Backup Server
Backs up various flat files as well as block-level servers and PC backups
Syncs to the cloud and allows local and cloud virtualizations
Blue Iris
NVR to capture all of my video surveillance around my house
Utilizes CodeProject.ai for various recognitions including people, vehicles, license plates, animals, specific deliveries (e.g. Amazon, UPS, FedEx), etc.
February 2023 Update:
After receiving all of the new hardware, I had a lot to set up. The first step thing to do was to clear out my existing lab and start making room for the new stuff. I went and picked up my server rack, which i found on Facebook marketplace, and got it into my basement. I racked the patch panel, the TP-Link switch, and the CyberPower UPS and surge protector first.
I then set up VMware ESX on my R430 with the drives from my old main server as locally attached storage. Unfortunately, I ran into problems here. Whenever I attached more than just my SSD to the server, the R430 was throwing "health" and "electrical" indicator errors, and the server wouldn't POST. iDRAC was showing CPU voltage misalignment. In the end, I had to return this for a replacement server.
I also converted my Blue Iris tower to the Rosewill rackmounted chassis. In doing this, I decided to change out the current mini-ITX motherboard for a full size ATX board. This gave me more options for the future. If I wanted to convert this server into another ESX host, I would be able to add several networking interfaces via the PCIe cards, and I would also be able to upgrade the RAM with the additional 2 DIMM slots. Plus this board had a 2.5 Gbe NIC and several M.2 NVMe slots available on it, versus the single gigabit and single M.2 slot on the old board. I used all of the old hardware from the tower though, and threw in some Noctua fans I had laying around for the intake, and NZXT and Cooler Master fans to replace the Rosewill fans in the middle row. I also ended up 3D printing an SSD adapter to fit in one of the HDD bays.
Another problem I unfortunately ran into was that the official Rosewill rails and the recommended 3rd party StartechUSA rails both would not work for my use case without modification, as they use M4 screws and not 10-32 screws like my rack uses. My rack door would also not fully shut with the Rosewill handles attached. In retrospect, I regret not going with a Sliger case. To remedy these issues, I had to remove the rack mounted ears on the Rosewill chassis and have it sit on a Dell shelf that I had. This isn't a terrible solution, but it does use an extra 1U of space, isn't secured directly to the rack, and won't give me the ability to slide the server out more than a few inches to work on it.
After the initial hardware was all set, I installed an additional outlet in the area on a new circuit to make sure I wasn't overloading anything. I had a different outlet installed nearby for my homelab when I first moved in, so I now had power for my homelab on two different circuits which was nice. I also have two UPS's set up, one on each power outlet, for some redundancy.
The next step was to install the exhaust fan where all of the hot air would be accumulating towards the back of the server rack. The fan is on a thermostatic controller and pushes the air from under the stairs into the basement, which generally stays pretty cool throughout the year.
For this part of the project, I found an STL that someone created to rackmount the LCD display for the fan controller. I had a different model with a different size display though, the "67 Controller", so I modded the STL to the proper size while adding blocks and pilot holes for the new screw patterns and blanking out the old screw holes.
Unfortunately, I ended up going with a different fan, their 10" version, which included yet another controller (69 Pro Controller) with a different LCD size, so I had to design and print yet another design. All of these can be found on my Thingiverse profile.
Now that it was printed and looking good, it was time to install the fan into the wall and mount the controller LCD in the rack. The fan has an app that connects to WiFi that lets you set temperature and humidity points for it to activate. The fan itself is really quiet and moves a ton of air and I'm really happy with it.
To wrap up the initial hardware in the rack, I added my two NAS boxes to a shelf on the bottom and got them hooked up. One would remain off for now while the other continues to be a backup device. I added my modem and router to a shelf towards the top and the rack was done for the time being. I also decided to mount one of my mini-PC's to the back of the monitor with a 3D printed VESA mount that I designed to hold it.
Both the rack itself and the overall homelab now were in a position that I was very happy with. It had been heavily upgraded and was also much more organized. I have some future plans that I will post when I expand on the lab, but for now I'm going to work on setting up the software side of things and learn new applications with it.
Future Plans
Dive deeper into the networking side of things. I would like to re-configure my entire network with VLANs, set up a (potentially virtual) pfSense router, and continue configuring virtual switches, LAGs and various setups of NIC teaming for the ESX host.
Multiple ESX hosts. Potentially convert the Blue Iris server or a NAS to an ESXi host.
Increased storage with a few different options:
Utilize an extra 4 bay NAS tower w/ 10 Gbe NIC with direct link to ESX host.
Purchase something like a NetApp DS4246 JBOD, get a SAS PCIE card, and MiniSAS cord and hook this up directly to the ESX host.
Set up a different less powerful ESX host, possibly on a NAS tower or the Blue Iris server and add local storage to it. Create a vSAN managed by vCenter.
Install Nextcloud or similar. Install various other self hosted systems
Continued learning with an Information Security focus as well as automation focus (Puppet, Ansible, Terraform, etc.)
Comments