Fun with PI Series - An ARM Dev Board Matrix That Looks Like a Server - Firefly Cluster Server

Introduction

Based on my work and personal interests, I wanted to set up a server cluster at home for testing containers, K8s, and related solutions.

I considered using a used server, such as a Dell R730, and even put together a spec sheet like this:

• Dell R730
• 3.5-inch form factor hard drives
• CPU: 2686v4 × 2
• Memory: 16 GB × 8
• Storage: 480 GB Intel SSD as system disk + 6 TB SAS Seagate × 2 as data disks
• RAID card: H730
• Power: Single 750 W PSU
• Full drive bays
• Price: approximately ¥3,130

Plus a server rack…

But considering the high power consumption and noise, and the fact that I didn’t have enough space at home for a full-size server, I gave up on that idea. 😂

I also considered single-board ARM dev boards, but that would mean getting several of them, then compiling the OS, flashing firmware, finding suitable cases, sourcing power cables and Ethernet cables, and connecting them to the home network… Setting them up one by one would take way too much time.

Even after getting everything working, I probably wouldn’t have enough Ethernet ports at home and would need to buy a switch…

In the end, persistence paid off. I found something on Xianyu (a second-hand marketplace) that combines the best of both form factors — an ARM dev board matrix that looks like a server — the Firefly Cluster Server. It fits my needs perfectly 👍️, let’s take a look!

Firefly Cluster Server - ARM Dev Board Matrix

Full Server Overview

Let’s start with an overview of the server:

It looks like a small form-factor server, but don’t be fooled by its appearance — it’s completely different from an x86 server on the inside.

This is a customized version based on the Firefly Cluster Server R1 — the core-3399-jd4 × 11 model, with fewer external ports. However, the internal architecture is identical.

This is the rear panel: 2 small fans + a power connector.

Let’s open the top cover and take a look inside:

The cooling design is: passive heatsinks on each CPU + active fan cooling for the entire chassis.

Besides the power supply, there’s a single mainboard (backplane). This mainboard is quite interesting — it consists of:

• 11 edge connector slots for inserting Firefly core-3399-jd4 compute modules, one per slot. That means there are 11 compute modules in total, corresponding to 11 ARM servers. I’ll cover the per-server specs later.
  • 1 is the main core, serving as the management node — similar to a server’s BMC — to manage the other 10 servers.
  • The other 10 are worker nodes for running actual workloads.
• A switch chip for internal Ethernet communication between the 11 compute modules.
• 4 Ethernet ports:
  • 1 for the main core
  • The other 3 are connected to the backplane’s switch chip

Here’s the network topology diagram, which gives a very intuitive view of the server’s internal architecture:

Server Core - Firefly core-3399-jd4

This is the core of the server: a hands-on photo of the Firefly core-3399-jd4. It fits in one hand — quite small:

Each Firefly core-3399-jd4 compute module is essentially an ARM server with the following specs:

• CPU: Rockchip RK3399 hexa-core ARM64 CPU (28 nm process — it runs quite hot)
• Memory: LPDDR4, 2 GB / 4 GB options
• NPU: Optional — not included in my purchase
• Storage: eMMC, 8 GB – 128 GB options; mine is 32 GB

In other words, once this server boots up, you get 11 machines, each with 6 cores, 2/4 GB of RAM, and 32 GB of storage.

Taking the 4 GB / 32 GB configuration as an example, the totals are:

• 66 CPU cores
• 44 GB RAM
• 352 GB storage

Flashing Firmware

This type of server was originally designed for cloud phone farms, so it ships with Android by default.

For my use case, I flashed all boards to Ubuntu.

I won’t go into the detailed flashing process here. If you’re interested, check out:

Sub-board Firmware Upgrade - Product Documentation — Firefly Wiki (t-firefly.com)

Main Core - BMC Management Interface

The BMC management interface of this server is also quite interesting. Its main features include:

• Device list
• Device monitoring dashboard
• Shell terminal
• Sub-board firmware upgrade

As shown below:

Does this interface look familiar? 😄😄😄

It’s actually built on a modified version of Grafana.

• The UI is: Grafana + some custom panels
• Monitoring data is powered by: Prometheus + node exporter
• The shell terminal is implemented via ADB (📝 The 11 compute modules in this server are interconnected via the Ethernet switch; additionally, the 10 sub-boards are also connected to the main core as USB devices)
• Sub-board firmware upgrade is implemented using Rockchip’s Linux flashing tool

Hands-On

Here’s what the server looks like on its first boot:

After a lot of tinkering, 4 boards ended up dead 😂😂😂, leaving only 7 operational.

That hurts… 💀💀💀

Power Consumption

Power consumption is stable at 30–40 W (7 boards, roughly 5 W per board), as shown below:

The power draw is quite low — that’s the advantage of ARM chips.

Summary

Based on my work and personal interests, I wanted:

• Multiple servers
• Always-on operation
• Low noise
• Low power consumption

This machine looks like a server but is actually a matrix of 11 ARM dev boards, and it met my needs.

• After breaking 4 boards through tinkering, I still have 7 usable ones
• It can run continuously
• The noise is still a bit loud (mainly due to the aggressive fans — I’m considering replacing them with quieter ones later)
• Low power consumption

It largely meets my requirements. Next, I’ll use it to set up a HashiCorp Nomad cluster.

Stay tuned.

📚️ References

• CSR1-N10R3399 Downloads | Firefly | Making Technology Simpler, Making Life Smarter (t-firefly.com)
• Welcome to Cluster-Server-R1 Manual — Firefly Wiki (t-firefly.com)
• Welcome to Core-3399-JD4 Manual — Firefly Wiki (t-firefly.com)
• Core-3399-JD4 Downloads | Firefly | Making Technology Simpler, Making Life Smarter (t-firefly.com)