What is Bitcoin Mining?
Simply put, Bitcoin mining is the process of creating new Bitcoins by verifying the transactions in the
Bitcoin network. Today, this is mostly done using purpose-built Bitcoin mining devices which are used to solve a mathematical problem (hashing). Miners get rewarded with 25 new Bitcoins per new block in return. A new block is generated approximately every 10 minutes, resulting in a total of 1.3 million Bitcoins or $845 million per year at current valuation.
The cost of verifying transactions is the capital cost to buy miners, the power consumption of running the hardware, plus operational expenses to keep the operation running. The more computational power is employed to do the “hashing”, the bigger the share of the total reward that goes to the miner. To stabilize the block creation rate at one block about every 10 minutes, the network selfadjusts the difficulty of the hashing calculations. As a result, the rate of creating new Bitcoins stays the same, no matter if there are 100 or 100,000 miners. 1
Industrial-Scale Bitcoin Mining of Today
While Bitcoin mining initially used to be carried out at home, in basements and in makeshift "mining farms", we see an increased move to industrial-scale data center mining by investors, mining groups, cloud mining providers and device manufacturers.
One such manufacturer is currently building a 10MW data center next to Facebook in Sweden 2, and there are currently more than a dozen known manufacturers engaged in developing ASIC chips for
Bitcoin or other cryptocurrencies, including in the USA, Russia, Europe and China. All this new hardware will add additional resources which have to be powered.
During the past 24 months, the networks total hash rate has grown exponentially. Despite increased hashing device efficiency (from GPU based mining to 28nm ASIC based mining) the global power required for mining-devices has grown 147 times in the past year alone. The power that supports the current Bitcoin network is estimated to be 250-500MW 3, and will double or could even triple during the next 12 months just from one hardware manufacturer alone. 4
Cooling energy, the power required to keep mining devices and mining farms cool, is estimated to account for an additional 30-50% on power consumption globally. It is this component of the overall electricity need that is the most variable and also the most addressable as different technologies can be implemented to reduce the cooling electricity overhead substantially.
Figure 1: 24 Month Growth of Bitcoin Total Network Hashrate
Example #1: Cointerra TerraMiner IV Mining Cluster
The TerraMiner IV (manufactured and sold by Cointerra Inc in Texas) is presently one of the popular devices and the company claims to power 6% of the entire Bitcoin network. It comes in the form factor of a 4U enclosure and uses four 28nm ASIC chips with 500W each. The chips are water cooled with a server grade water cooling system consisting of water blocks, 2 pumps, 2 custom radiators, 5 cooling fans in the enclosure and additional fans in the two power supplies. Total power consumption is 2000W (2kW) at a performance of 1.6TH/s (Terrahashes per second) and a price of $6000.
Figure 2: Cointerra TerraMiner IV
As of this writing, the total network hash rate is estimated to be 35PH/s (Petahash/s), which is the equivalent of around 21,875 TerraMiner IV units. By extension, one would have to commission 2,188 units worth $13 million to own 10% of the network, or 218 units worth $1.3 million to own 1%.
For a 1.2MW cluster with 600 units, at 10 units per rack, this implies a physical foot print of 1,200 / 2
/ 10 = 60 high density 20kW racks @ about 60 x (8x2) ft2 = 960 ft2 of data center space to own
2.74% of the network (with a hardware price tag of $3.6 million). All this would net the miner about
$60,800 per day in gross revenue at current valuations (~$643 USD/BTC). 5
Example #2: ASICMiner Gen3 - Most Efficient Next-Gen Cluster
The Generation-3 ASICMiner chip (manufactured and sold by ASICMiner/Bitfountain) is presently considered the most efficient and least expensive mining device to be released in 2014. The hardware will be available from ASICMiners partners and distributors worldwide, adding a total hashrate of 400PH/s to 1600PH/s to the network within this year 6.
The final price is currently unknown, but the chips themselves are sold for $500 to $1000 per TH/s.
Initial tests suggest a power consumption of 500W with a performance of 1TH/s and room for improvement. 7
A very conservative assumption would be a 50% increase in manufacturing cost for the finished board including power-train and power supplies, resulting in a final blade-style unit with 4TH/s and
2000W power consumption at a manufacturing price of $1500.
Assuming total network hashrate of an arbitrary 175 PH/s (5X current network) at the time of deployment, this would mean the equivalent of around 43,750 AM-Gen3-2000W units. By extension, one would have to commission 4,375 units worth $6.6 million to own 10% of the network, or 436 units worth $654,000 to own 1%.
Assuming the space usage equivalent of a 4U enclosure for sufficient air flow, a 1.2MW cluster with
600 units, at 10 units per rack, this would imply a physical foot print of 1,200 / 2 / 10 = 60 high density 20kW racks @ about 60 x (8x2) ft2 = 960 ft2 of data center space to own 1.37% of the network (at a hardware cost of $900,000).
All this would provide the miner about $30,400 per day in gross revenue in an arbitrary 175PH/s future network hashrate (~$643 USD/BTC). 8
Challenges of Industrial Mining
Data Center Mining
60 cabinets of 20kW each result in a power density of 1250W/ft2, while typical data center space is usually not more than 120 W/ft2 or between 2-4kW per rack. 9 Each cabinet is usually assessed with
30 ft2 of required space 10 incl. cooling (ten times less the density of a fully populated Bitcoin mining rack), and data centers with a density of 1000W/ft2 cost 8X more to build than standard data centers 11. Furthermore, TCO costs (total cost of ownership) increases significantly after an optimum power density, which is in the order around 6kW per enclosure. 12
Using Digital Realty Trusts POD 13 Architecture™, which is 1.25MW for 10,000 ft2 and carry an uplift of 50% for cooling at any density, a 1,250KW load would handle 600 rigs in 60 racks. Digital Realty
Trust has a $146/kW all in cost for rent on multi-year leases with 75+ pages contracts to sign, plus location-dependent power costs (average of .10/kWh). This means that a tenant using a traditional data center will spend over $146,000 per month in rent and $131,400 for electricity for fully occupied space. Annually that is $3.8 million.
It is increasingly difficult to find facilities with the aggressive cooling systems that are required to cool
Bitcoin mining hardware, and miners are now competing with large businesses for data center wholesale space and infrastructure.
Data centers are not prepared for Bitcoin densities, and industry trends suggest the power density projections displayed in Figure 3 with extreme densities at 10kW per rack.
Figure 3: ASHRAE Power Trend Charts for Data Centers, maximum measured loads in watts from a fully configured and highly utilized rack. 14
9
Calculating Space and Power Density Requirements for Data Centers http://www.theicor.org/art/present/art/ARCIFM00024.pdf 11
Calculating Space and Power Density Requirements for Data Centers
12
Cooling Strategies for Ultra-High Density Racks
13
Digital Realty Powers Up its POD Architecture
14
Datacom Equipment Power Trends and Cooling Applications, Second Edition, ASHRAE Datacomm Series, Year 2012
10
Page 5
Purpose Built Large-Scale Mining Facilities
With multiple Megawatts becoming the norm, purpose-built "ghetto-mining" facilities with complex structures and large engineering and administrative overhead are no longer a good option. More professional solutions are required.
Imitating Google's and Facebook's open approach to server hardware and data center architecture
(Google Platform and Open Compute Project) looks like a good idea at first, but proves to be a time consuming process. Both companies spend hundreds of millions to build their data centers, and both build for hardware cycles of several years, not for under a year at best when mining.
In the rushed world of Bitcoin mining, a constant cycle of infrastructure upgrades and maintenance, deployment and re-deployment, tends to lead to overhead and delays, and the need for a growing number of staff to work 24/7s on a continuous basis. It's not the "wash-rinse-repeat" experience that one would expect from the business of Bitcoin mining, where long down-time to setup a new cluster leads to higher opportunity costs.
KnC Miner 10MW Facility 14,500m² (156,000 sq ft)
The KnC Miner Company has built a 3MW facility with 5500 square meters (59,000 sq ft) in Sweden and is adding another 9,000 square meters totaling 10MW before the summer. To keep the miners running, KnC employs 30 people at the moment. 15
Very little is known about KNCs cooling method, but given the cold climate of Sweden it is assumed to be "free air cooling" without cold or hot aisle separation. To efficiently remove all the heat from the building, the air flow required is 75 million CFM (600.000 L/s) for 10MW, in addition to the thousands of device fans on mining devices. 16
KnC does not use cases on their mining units, but the bill of material includes thousands of device fans, heat sinks, heat pipes and thermal pads/paste, plus the associated labor overhead during production and maintenance to keep the cluster running. KnC's current hardware uses 4 fans if used without case, resulting in tens of thousands of fans at full deployment of 10MW.
Once completed, the facility will have a power density of 14,500 m² / 10,000kW = 1.45 m2 per 1kW of mining power, leaving very little flexibility towards higher density. Doubling the capacity with an identical foot print would result in 0.725 square meters per 1kW, assuming appropriate pre-planning to double rack/row/device power distribution from 10MW to 20MW. In the light that KnC is reportedly looking for additional locations, an increase in density is not very likely.
Figure 4: The KnC Miner Data Center in Sweden
MegaBigPower 1.4MW Mine 1900m² (20,000 sq ft)
The MegaBigPower Company focuses on Bitfury's hardware and has invested over $1 million into a custom 1.4MW facility in the USA. The mine initially started with 30 tons of air conditioning, and after citing problems, their fans now move 150,000 cubic feet of air per minute. During the cold winter at negative 1°C (30°F) on the outside, the inside of the mine is showing 39°C (102°F) on the
15
16
thermometer 17. Summer is expected to be a challenge. MegaBigPower estimates that the operation costs $1 million and now involves 15 people. 18
Very little other details are known, but an American TV station has recently shown the mine from the inside on a 5 minute video segment that can be seen on YouTube. 19
Figure 5: MegaBigPower Mine in the USA
ASICMiner 500kW Immersion Cooling 30m² (300 sq ft)
ASICMiner subcontracted building and running their immersion cooled 500kW Gen-1 ASIC cluster to the author’s company. The mine was built in 2013 in Hong Kong and serves as a proof-of-concept for a new era of high density computing. Electronics Cooling magazine has featured the mine on the cover page and an article titled "Bitcoin Mining and the Implications on High Density Computing".23
Boards that are normally designed for air cooling were stacked as dense as possible in sets of 92 pieces per tank. 20 racks, 60 tanks, 5,520 boards in total and 240 high density power supplies at full capacity. With no upper limit of cooling performance, the board-to-board distance could have been smaller but was dictated by the size of the Ethernet connectors.
The same fluid/tanks can accommodate 75kW per rack with current condensers, or up to 225kW per rack with condenser upgrades. The footprint of the system is less than a standard shipping container, including all peripheral gear such as outdoor dry cooler or power distribution.
However, the mine was installed into a brick and mortar facility and is limited by the 500kW power the building can provide, with only 60A 3-phase power per tank. Deployment of Generation-3 hardware is currently being prepared and can be accompanied up to the 500kW power constraint.
This mine cannot be relocated without major building works, and should more power become available, the rack level power distribution will need an upgrade. The cost of initial build-out of electricity and water loop account for over $100,000, which would have to be re-invested on a relocation, in addition to months of build-out time and the labor intensive job of getting it done.
Sufficient power supplies for Gen-3 deployment have already been acquired during installation of
Gen-1. ASICMiner never produced Gen-2 chips, further illustrating the need for flexibility in any industrial mining environment and pre-planning for large capacities.
In early 2013 when the project was initiated and the first ASIC chip was just released, 500kW seemed like a giant leap forward. The company had previously commissioned an immersion cooled 64kW
FPGA cluster in 2012.
The cluster is fully automated and runs silent and dust free, with no moving parts. Significantly reduced maintenance overhead requires only a single person to check on the system every few days, mostly via networked reporting tools. No single system failure has been recorded to date on the cooling system. A Roomba iRobot keeps the mining facility clean, and the freed up space is used as meeting room and for presentations.
Figure 6: ASICMiner Immersion Cooling Cluster in Hong Kong
250kW IceDrill Mine
Another prominent example of a commercial operator is the IceDrill Bitcoin mine. The investment involved building "a secure and climate controlled mine-facility" including "build-out of computecenter/mine location" in Canada. 20
The company has mining-devices from HashFast on back-order and additionally invested into further hardware from Cointerra, ultimately leading to the same data center requirements outlined in above data center example. IceDrill is effectively trying to host a large number of retail boxes that require two separate power inputs in a data center environment that is not prepared for Bitcoin densities.
USD $285,000 was spent on the data center facility 21 so far, with 120 mining devices 22 deployed bringing the total capacity to 144kW at the time of writing, and the cost to $1,979 per kW.
Larger Facilities, Larger Investments, Higher Risks
With rising Bitcoin popularity it can be expected that investments into Bitcoin mining will remain attractive for a long time. The disruptive nature of bitcoin for many existing businesses and even monetary schemes make its valuation highly volatile within the foreseeable future, and new Bitcoins are created until 2040 by design.
Just as Bitcoin is driving innovation, it is also pushing the limits of hardware manufacturing, deployment, cooling, power distribution, infrastructure, connectivity and administration. With higher densities come high and unpredictable costs that vary with every data center operator. Bitcoin mining hardware density has long exceeded common IT hardware and even supercomputer densities 23.
Renting and building out 10MW sites comes with a complete new set of implications. Lease terms, businesses and budgets involved cannot be described as simple. 5-10 year lease agreements with
75+ pages and prices based on credit checks, are not unusual. In contrast, renting smaller industrial facilities with 1-2MW or less appears simple, often only requiring industrial space and the electricity company to establish power.
With larger investments being required and a maturing Bitcoin market, it can be expected that more traditional investors will join the arms race for the fastest and largest Bitcoin clusters. New investments from countries like China and Russia are adding to the pressure.
Shorter Lifespan, More Resources, More Points of Failure
Decreasing lifespan of mining hardware and a steep increase in network difficulty leads to the need for constant upgrades and new investments. Power and thermal design, supply chain, administrative and logistical overhead is taking up significant resources and leads to unnecessary delays. Cooling system parts including heatsinks, fans, water blocks, and support hardware such as enclosures, power supplies and mechanical parts are not an essential part of the mining core, but very often take
20
up a majority of the manufacturers or mine operators resources. This results in a large number of suppliers that must be orchestrated to deliver a working product.
Larger facilities need more time to adapt their cooling and power systems to support the next hardware generation, and will make this orchestration even more difficult. Thus, resulting in an inflexible operation where the operator needs to rely on many third parties to deliver a working and efficiently running operation.
Cloud mining can eliminate this problem for the small-scale miner, but shifts the need to deal with these challenges to the cloud operators. “Traditional” mining methods and brick and mortar data centers seem unable to provide the flexibility that Bitcoin requires during fast hardware cycles.
Thermal Issues at Board Level
Keeping hardware cool has proven to be a problem right from the early days of Bitcoin mining.
Devices automatically down-clock when overheating, and even the manufacturers of the latest and greatest mining devices are forced to employ throttling techniques or sell hardware below advertised specs, due to thermal issues with the chips or power-trains.23
As chip manufacturers are approaching the limits of traditional cooling methods, today's high powered ASIC devices face a new problem altogether. Bitcoin mining devices are exceeding heat dissipated by commercially available computer hardware, creating a problem to find simple and economic cooling solutions.
Proposed Technical Solution
1.2MW Bitcoin Mining Container
The next-generation immersion cooled container mining units are compact and dense modular total system solutions, including cooling, power distribution, communication, automated monitoring and reporting. Container mining units consist of six 200-240kW flat-rack tanks and systems are designed to work with any hardware on the market, including current and future ASIC boards or GPU clusters.
Except the actual mining boards, nothing needs to change when the next generation chips are released. With no thermal constraints and no fans or water blocks, manufacturers can simply change the size of existing or future boards to make best use of available space. A few millimeters of fluid between chips and boards are usually enough.
With efficient and solid integrated 12V power distribution, board manufacturers do not need to worry or buy new power supplies with each and every generation.
The operations plan is centered on simplicity and scale, and to deliver a wash-rinse-repeat experience for mining operations and hardware manufacturers alike.
System Highlights
●
●
●
●
●
●
●
●
●
Complete 1.2MW turnkey mining solution
Less than 1% power used for cooling, even in the most demanding climates
Lowest build cost per kW compared to any other method
Redundant systems, including cooling and networking
Modular and universal design, fits any hardware
Mining boards simply slide in and out of backplane slots
Integrated 19-Inch rails, compatible to GPU or HPC systems
Silent and dust free operation, no moving parts
There is no need for cables or power connectors
Page 9
●
●
●
●
●
●
Fully automated system monitoring & reporting (HTTP, SNMP, Modbus)
Integrated security and remote HD camera monitoring
Fluids are non-flammable, non-toxic, environmentally friendly
Manufactured in the USA, all equipment conforms to UL and CE standards
Works with wide range of power feed voltages internationally
Solid power distribution with room to grow (per tank max 800A 3-phase)
(the equivalent of the total 500kW AM system in Hong Kong per tank)
6 tanks per container, 200kW-240kW each
400 boards each tank, 500W per board, 2,400 boards total
Equivalent to 600 4U boxes
Shipping container footprint instead of 60 racks in high density facility
2-3PH/s at estimated hashrate (based on reports from China 24)
No thermal throttling, possible performance increase to be tested with actual hardware
Fixed and predictable cost, no matter if fully or only partially populated
High Density 19-Inch Slots
Included with the tanks are simple universal slots that can hold 1-6 power supplies and fit any mining device on the market. For optimal density, custom boards allow various install options. Slots slide into tanks without any tools, screws or nuts. If tanks are not fully populated, spacers can be inserted from the top to save fluid (ie. for partial deployment).
Standard 19-inch rails allow installation of standard IT equipment such as HPC or GPU clusters. Time tested Titanium efficiency high-density power supplies with a total of 4-4.8kW output power per slot are recommended for Bitcoin mining and can be supplied as pre-installed option in tanks.
Figure 7: Example of universal 19-Inch slots and backplanes included for Bitcoin mining
(final specifications currently being discussed with hardware manufacturers)
Fully Owned or Managed
Fully owned containers can be used for remote deployment. Sufficient training, documentation and
24
hands-on assistance will be provided in order to start operations.
Managed solutions are being looked at with two different strategically located 10MW facilities throughout the USA under consideration, to further increasing benefits for key-partners. Location scouting in the USA has commenced in January and can be adjusted depending on demand. Other locations, business models and partnerships can be considered as well, including exclusivity.
The company is currently looking into forming key partnerships with mine-operators and manufacturers in order to finalize specification and requirements. The aim is to produce a small series of containers throughout 2014 and establish one or two facilities, leading up to full series production by beginning of 2015.
Figure 8: 1.2MW immersion cooled mining cluster with 6 tanks 200kW each tank
(overhead busway power-feed not depicted)
Conclusion
Industrial Bitcoin mining comes with a high price, overhead and limitations. When you take into consideration build-out, lease, operation and frequent hardware upgrades, it's not very hard to figure out that making money with industrial mining can be very costly. Being involved with creating mining devices adds further complexity.
While in 2013 a few hundred kilowatts or 1MW was considered big, Bitcoin miners are now looking at blocks of 10MW or more, and are competing for data center space with well-established IT businesses with high credit scores and whose data center density requirements will never be at the low end of Bitcoin densities. With a maturing Bitcoin ecosystem, new investors are entering the market and add more pressure.
Modular container sized mining units can be the most efficient way forward. Hardware can be reduced to chips on boards and can be produced without wasting resources on infrastructure, power, enclosures, thermal designs or logistics. Hardware can go from chip design to board design to mining weeks or months earlier and cost less to produce. A true wash-rinse-repeat experience can be established. Industrial sites with power-feeds of "only" 1.2MW are easier to come by than large-scale data center buildings which are under high demand by the data center industry. Decentralized deployment presents a safe and economical way to ensure safe operation. Container units can accompany any size of cluster, from smaller 1.2MW sites to large-scale installations the size of Google's 45 container data center in Oregon.
Containerized and modular mining systems can be moved if required and sold on the secondary market for liquidation or ownership transfer. With Bitcoin on the rise and hundreds of potential crypto currencies and new ASIC based Scrypt Miners on the horizon, there will be more demand for a highly efficient industrial mining solution.
Page 11
