ORIGINAL RESEARCH

FORECASTING
SIGNIFICANT
SOCIETAL EVENTS
USING THE EMBERS
STREAMING PREDICTIVE
ANALYTICS SYSTEM
Andy Doyle,1 Graham Katz,1 Kristen Summers,1
Chris Ackermann,1 Ilya Zavorin,1 Zunsik Lim,1
Sathappan Muthiah,2 Patrick Butler,2 Nathan Self,2
Liang Zhao,2 Chang-Tien Lu,2 Rupinder Paul Khandpur,2
Youssef Fayed,3 and Naren Ramakrishnan2

Abstract
Developed under the Intelligence Advanced Research Project Activity Open Source Indicators program, Early
Model Based Event Recognition using Surrogates (EMBERS) is a large-scale big data analytics system for forecasting significant societal events, such as civil unrest events on the basis of continuous, automated analysis of large volumes of publicly available data. It has been operational since November 2012 and delivers approximately 50 predictions each day for countries of Latin America. EMBERS is built on a streaming, scalable, loosely coupled, shared-nothing architecture using ZeroMQ as its messaging backbone and JSON as its wire data format. It is deployed on Amazon Web Services using an entirely automated deployment process. We describe the architecture of the system, some of the design tradeoffs encountered during development, and specifics of the machine learning models underlying EMBERS. We also present a detailed prospective evaluation of EMBERS in forecasting significant societal events in the past 2 years.
Introduction
Anticipatory intelligence is considered to be one of the next frontiers of ‘‘big data’’ research, wherein myriad data streams are fused together to generate predictions of critical societal events. One of the promising themes in this space is the idea of harnessing open-source datasets to identify threats and support decision making for national security, law enforcement, and intelligence missions. Early Model Based
Event Recognition using Surrogates