Privacy Snooper: IOT
Arnab Kumar1 , Harishma Dayanidhi1 and Vijay Kumar KS1
{arnabk, hdayanid, vkanlanji}@andrew.cmu.edu
1

Carnegie Mellon School of Computer Science, Pittsburgh, USA

Abstract. In various ML-as-a-service cloud systems, the process of performing machine learning over the data is almost treated as a black box, where the user just feeds in their data, knows the model used and the system outputs required insights. In this work, we explore the idea of being able to predict sensitive attributes associated with the database given that the adversary would have access to a few quasi-identifiers associated with the database. We use inversion attack as the theoretical foundation for our attack, and implement the same for our database. We experiment this attack for di↵erent variants of classification algorithms, like classification tree and regression tree. We follow it up with analysing the accuracy of our attack for each of our classification based machine learning algorithms for di↵erent size of training datasets. We end our work by trying to figure out what we say is the ”most impactful attribute”, by selectively removing the data pertaining to an attribute and check what is the corresponding e↵ect on inversion attack. We hope our work in this domain pushes future batches of this class to explore this question even further, and too look into understanding if Di↵erential Privacy solves this problem.
Keywords: Inversion Attack, Black Box, Classification Tree, Regression Tree

1

Introduction

The Internet of Things (IoT) is the network of physical objects or “things” embedded with electronics, software, sensors, and network connectivity enabling objects to collect and exchange data. British entrepreneur
Kevin Ashton first coined the term in 1999 while working at Auto-ID Labs (originally called Auto-ID centers
- referring to a global network of RFID connected objects. The bounds of such an inter-connected world is truly mind-blowing. The Internet of Things allows objects to be sensed and controlled remotely across existing network infrastructure, creating opportunities for more direct integration between the physical world and computer-based systems. Such an inter-connected network of devices resulting in improved e ciency, accuracy and economic benefit. Experts estimate that the IoT will consist of almost 50 billion objects by
2020.
The term IoT has been used to refer not just to the advanced connection of devices, typically machine-tomachine communications (M2M) but also a wide variety of protocols, domains, and applications. “Thing”, in the IoT sense, can refer to a wide variety of devices such as heart monitoring implants, biochip transponders on farm animals, electric clams in coastal waters, automobiles with built-in sensors, or field operation devices that assist firefighters in search and rescue operations. These devices collect useful data with the help of various existing technologies and then autonomously flow the data between other devices. Current market examples include smart thermostat systems and washer/dryers that use Wi-Fi for remote monitoring.
The growing influence of IoT has now promulgated to surveillance devices as well. Surveillance is the monitoring of the behaviour, activities, or other changing information, usually of people for the purpose of influencing, managing, directing, or protecting them. This can include observation from a distance by means of electronic equipment (such as CCTV cameras), or interception of electronically transmitted information
(such as Internet tra c or phone calls); and it can include simple, relatively no- or low-technology methods such as human intelligence agents and postal interception. The wide variety of information collected over di↵erent types of surveillance ranges from video and audio to heat-sensing and biometric these days. Such surveillance by the government is justifies saying that it is used for intelligence gathering, the prevention of crime, the protection of a process, person, group or object, or for the investigation of crime. Such concerns have further accentuated given 9/11 and other terrorist attacks in recent times.
Such continual surveillance, by public and private entities alike, has raised several privacy concerns. Add to that the plethora of new application areas for Internet has caused previously incomprehensible amounts of data being generated. The development of sophisticated machine learning algorithms have now allowed extensive analysis of this data, enabling insights that blatantly invade into the privacy of individuals whose

2

A. Kumar, H. Dayanidhi and V. Kumar KS

data is collected, without user choice or notice of such data collection taking place. As students of Privacy Engineering, and having faced such privacy harms in our life, we felt that a solution for this is highly imperative.
One of the most obvious and foolish solutions in mind would be to ask the government and private companies to stop all forms of surveillance. Several public advocacy organisations, companies from across the political spectrum and initiatives like StopWatching.us have asked the government to completely cut down on surveillance. However, we believe that such a solution would not just be impractical to be implemented, but would pose huge concerns to national safety and security. After giving extensive thought to several probable solutions to this problem, we believed that perhaps the best possible way of going about is to properly notify an individual of the existence of all such surveillance devices. How individuals react to and after proceed such circumstances are at their own discretion. We aspired to come up with a solution that would give them the ability to be well informed of the existence of surveillance devices in their surrounding, and their type (like audio, video, biometric etc) so that an individual may react accordingly. They might also choose to avoid the route consisting of such surveillance devices if they deem it uncomfortable.
Our project aspires to firstly understand this intuition, and providing an introduction to inversion attack in Section 3. We then proceed onto extending the inversion attack for di↵erent classification based algorithms
- classification tree and regression tree, and lay the foundation for us to understand which classification based machine learning algorithm makes them more susceptible to inversion attacks in Section 3. We then describe the dataset used for the purpose of our research in Section 4. Section 5 explores application of
Inversion attack to our dataset, understanding the variation in the accuracy of the inversion attack for di↵erent sets of training sizes of the database, and trying to compute the most optimal training set. We also try understanding impact of di↵erent attributes have on the inversion algorithm, investigating how presence/absence would increase/decrease the probability of accurately predicting the sensitive attribute.
We then present some highlighted observations and the conclusion to our work in Section 6. We wrap up our work by proposing avenues of future work and a few lines of thankfulness for our mentors in Section 7 and Acknowledgements respectively.

2

Related Work and Literature Review

In our initial proposal for the semester project, we proposed an idea incorporating two ideas of secure machine learning: Di↵erential Privacy and Machine Leaning on encrypted databases. We believed that incorporating these two seemingly orthogonal problems would give us what we believed complete security for machine learning. Our idea aimed to address privacy concerns protecting data contributors from adversaries keeping di↵erent threat models in mind: where the adversary is the analyst (di↵erential privacy) or the case where the adversary is a person trying to access the original database (machine learning on encrypted databases).
Streamlining our idea required us to read variety of papers, both from the perspective of how machine learning could leak information about the model being used and if combining this idea with Nina Taft’s work on Machine Learning in encrypted databases [?] would be practical for the purpose of this project. Sifting through multiple papers on both these topics enabled us to arrive at the conclusion that it would pragmatic for us to concentrate on the former: Understanding how machine learning models could leak information.
This led us to look into the works by Dr. Fredrikson on Inversion Attacks enabling an individual to backtrack and predict sensitive attributes associated with an individual from partial knowledge of quasi identifiers and the resultant of the machine learning model [?] [?]. A basic introduction to the idea around Inversion Attacks is defined in the next section.
The maturity of work done around the idea of staging inversion attacks on machine learning is still in it’s nascent stages. The paper [?] shows how machine learning has now enabled doctors to predict the recommended dosage of warfarin dependent on a sensitive attribute like genetic information and other quasi identifiers like geo-information. However, the authors then proceed onto proposing a black box inversion attack that enables an attacker to predict the genetic information of an individual in the database from his/her quasi-identifiers and recommended dosage of warfarin. What was most appalling about this work was the accuracy with which a highly sensitive attribute like the genetic data could be predicted. Such methods were then extended in the paper [?], where the authors have extended the algorithm developed in
[?] for cases in facial recognition etc.

3

Sequence

With a topic as broad as Privacy in the realm of Internet of Things, the journey from initial idea conceptualisation to final implementation involved introspection and intensive research at each and every step.

Privacy Snooper: IOT

3

This section guides the reader through each of these steps, and clearly elucidates every decision taken by us, along with the corresponding reason for doing so.
3.1

Idea Conceptualisation

With severe security concerns prevailing worldwide, continual surveillance by public and private entities has emerged as perhaps the only solution to counter, or at least provide su cient evidence for the investigation of such problems. This solution, though, has raised several privacy concerns. The combination of previously incomprehensible amounts of data and sophisticated machine learning algorithms have now led to extensive yet highly privacy invasive analysis of this data. Perhaps the scariest aspect of all this is that neither choice nor notice is provided to data providers when such data collection does occur. As students of Privacy Engineering, we gave extensive thought to several probable solutions to this problem.
Some contemplation led us to believe that perhaps the best yet simplest possible way of going about is to properly notify the user of the existence of surveillance devices around him/her. This, according to our expectations, would lead to individuals to become more “privacy aware” at the least, without taking any responsibility of how individuals react to surveillance notifications. We shall illustrate two of our most important ideas:

1. To build a static map based application that presents the user with the location of all surveillance devices. This would however require initial crowdsourcing by several users to identify all surveillance devices even within the perimeter of the university.
2. To build an application that could dynamically detect the presence of surveillance devices and notify the user about the same via standard notifications. Notifications could be customised for di↵erent types of surveillance e.g. Audio, Video and Biometric. Further detailed information about the surveillance could be addressed by a “Privacy Statement” associated with these notifications. The Privacy Statement is a twist on the conventional privacy policy, as it addresses questions regarding the surveillance that the users of the application found most useful. This is done in a manner that users find concise and easily understandable. Though this would require a mechanism by which surveillance devices could communicate with our mobile devices, it could be replicated via beacons positioned alongside these surveillance devices.
After a lot of thought put into each idea, and discussion with several PhD students at Cylab, we narrowed our focus on build an application that could dynamically detect the presence of surveillance devices and notify the user about the same via standard notifications. Some of the key points that convinced us to work on an iOS application for the latter are as follows:
1. The scenario associated with the latter idea is something we could see becoming the norm in the next decade. Though current surveillance devices do not posses the capability to communicate with other devices, we could easily replicate the same in a university or college setting via beacons positioned alongside them.
2. It represented a dynamic scenario of being notified whenever a person nears a surveillance device. This would have much more utility than the scenario of a person having to check out a map or continuously paying attention to it while he/she traverses the campus.
3. It enabled us to address another one of the broken systems in privacy and security these days: Privacy
Policy. Most of the privacy policies associated with data collection are extremely long and convoluted.
The “Privacy Statement” provided within our application is concise, to the point and addresses questions that users find most concerned about.
3.2

Basic Technical Decisions

Once the foundational idea for the project is set, we move towards illustrating the nuances of how to implement the idea for the purpose of this project. This would involve a variety of decisions, from choosing whether we would want to develop an Android/iOS application, to figuring out the beacons we should use.
All of our decisions have been taken with not just the technical capabilities of our team in mind, but also other factors that might a↵ect the scalability of this application for wider use in the future:

4

A. Kumar, H. Dayanidhi and V. Kumar KS

Operating System We chose to make an iOS application for the purpose of this application. The reasons for this choice are as follows:
1. Several studies like *cite* have shown that iOS user are more concerned about security and privacy than
Android users. This fits in well with the initial demographic that we target, which would be privacy savvy individuals.
2. The documentation provided by Apple for iOS application development is highly detailed with a highly active developer network and online presence.
3. Our team possessed more experience with developing applications on the iOS platform.

iBeacon OR Eddystone As mentioned earlier, our application would require a mechanism which would allow data collection devices such as video camera, audio recorder etc. to transmit information about them. with our mobile devices. Unfortunately, surveillance devices do not posses this ability currently. This would require us to validate the utility of our app through a workaround. We plan to do the same via Bluetooth
Low Energy (B.L.E.) devices like beacons accompanying all the surveillance devices.
Bluetooth is a wireless technology standard for exchanging data over short distances. This technology was developed by Ericsson Corporation in 1994. It allows a range of devices to speak to each other using radio waves. Bluetooth was designed to eliminate the need for connecting computer peripheral devices such as mouse, keyboard etc. with wires. Since its inception it has become a standard for transmitting information in the form of short range radio waves over distances of up to 30 feet and is used to connect and exchange information between digital devices.
Bluetooth Low Energy is a wireless personal area network technology used for transmitting data over short distances. As the name implies, it is designed for low energy consumption and cost. More details about
B.L.E can be found in Bluetooth 4.0 core specification [1]. B.L.E. maintains a communication range similar to that of its predecessor, Classic Bluetooth [2]. All major mobile operating systems including iOS, Android,
Blackberry etc. support B.L.E. technology. It finds varied applications, ranging from basic Heath care to
Sports, fitness etc.
Di↵erent vendors have di↵erent implementations of Bluetooth 4.0 communication protocol. For instance,
Apple Inc. has come up with iBeacon. Similarly, Google has come up with Eddystone protocol. iBeacon is o cially supported by iOS devices device only. However, Eddystone has support for both iOS and Android.
One major limitation of iBeacon is that there is no way a beacon can send some useful data back to the nearest mobile device. The iBeacon hardware transmits data that is used by the device to determine proximity to a unique iBeacon hardware device. iBeacon does not provide any mechanism to transmit custom data from the iBeacon to the mobile application. This seriously limits the utility of the iBeacon. Di↵erent companies have built beacons adhering to these standards. In our implementation, we explored a variety of
Bluetooth beacons. Some examples of the beacons that we explored are GemTot, XY Find It, BlueCats,
Estimote etc. Out of these, we selected Estimote for the following reasons: Estimote uses Eddystone. This means both Android and iOS devices are supported. Using Estimote, we can send a U.R.L. back to the mobile device. This is an advantage. Good documentation and active support for developers in the Estmote forum. Estimote beacons are easy to use and can be configured using a mobile device.
We aspired to come up with a solution that would give them the ability to be well informed of the existence of surveillance devices in their surrounding, and their type (like audio, video, biometric etc) so that an individual may react accordingly. They might also choose to avoid the route consisting of such surveillance devices if they deem it uncomfortable.
Model Inversion attack was first described in a case study of linear classifiers in personalised medicine by Fredrikson et al. The basic idea behind the attack is that an adversary with access to a machine learning model abuses this privilege to learn sensitive information about individuals, given that the adversary would be aware of a few quasi-identifiers associated with the individual whose data he/she is trying to compromise.
To explain the concept, we shall be taking an example of the warfarin dosage case study as discussed by
Fredrikson et al. in [?].
In our project, we assume an adversary who employs an inference algorithm A to discover the genotype of a target individual ↵. The adversary has access to a linear model f trained over a dataset D drawn i.i.d. from an unknown prior distribution p. D has domain XY, where X = X1 ,...,Xd is the domain of possible attributes and Y is the domain of the response. ↵ is represented by a single row in D, (x↵ , y↵ ), and the attribute learned by the adversary is referred to as the target attribute xt ↵ .

Privacy Snooper: IOT

5

In addition to f , the adversary has access to marginals p 1,...,d ,y of the joint prior p, the dataset domain
XY, ↵’s stable dosage y↵ of warfarin, some information about f ’s performance, and either of the following subsets x↵ K of ↵’s attributes:
1. Basic demographics: a subset of ↵’s demographic data, including age (binned into eight groups by the
IWPC), race, height, and weight. Note that this corresponds to a subset age race of the non-genetic attributes in D.
2. All background: all of p’s attributes except the genotype. The adversary has black-box access to textitf.
Unless it is clear from the context, we will specify whether f is the output of a DP mechanism, and which type of back- ground information is available.
The author went about trying to device algorithms for inferring genotype from a model designed to predict warfarin dosing. Based on the known priors, and how well the model’s output on a given row coincides with A’s known response value, the candidate rows are weighted. The target attribute with the greatest weight, computed by marginalising the other attributes, is returned.A mathematical formulation of the algorithm is given in [?]. They have derived each step by showing how to compute the least biased estimate of the target attribute’s likelihood, which the model inversion algorithm maximised to form a prediction.
For the purpose of the documentation, we shall not focus much on the mathematical foundation around which the black box model of the inversion attack is built, since it is not our primary contribution. All readers enthusiastic about reading on the same are recommended to read the paper [?] by Fredrikson et al., where the authors go to great lengths to explain the mathematical intuition and formulation of the concept. Our code for implementing the black box inversion attack has been attached along with our submission. As is expected, we shall be laying more focus on the results we obtained using various machine learning algorithms and the black box inversion attack to invert it.

Algorithm 1: Algorithm for Inversion Attack

1
2
3
4
5
6
7
8
9
10
11
12

4

Input: A Dataset with one or more sensitive attributes
Result: A graph for accuracy of the attack for a given begin l=0; for Each Dataset do for Each Classification based algorithm in set - classification tree, regression tree do
Pick a split for training and testing sets; for Every training split and testing set 1 do Train and acquire the model; for Each row in dataset do
Given the model and output value for a row, Run inversion attack on the model by substituting all possible values for the sensitive attribute and find its accuracy;
Find average accuracy of the inversion attack over all the rows;
Plot a graph for accuracy of the attack for a given;
Compare split values of highest inversion attack accuracy for each of the algorithms;

Dataset Used

The dataset used for the purpose of this work is a dataset named steak-risk-survey.csv downloaded from fivethirtyeight.com [?]. Participants in this survey were asked questions about their preferences for steak preparation. The survey also involved participants answering some personal questions. Following are the exact questions that were asked in the survey.
1. Consider the following hypothetical situations: In Lottery A, you have a 50 percent chance of success, with a payout of 100 dollars. In Lottery B, you have a 90 percent chance of success, with a payout of 20 dollars. Assuming you have 10 to bet, would you play Lottery A or Lottery B?
2. Do you ever smoke cigarettes?
3. Do you ever drink alcohol?
4. Have you ever been skydiving?

6

5.
6.
7.
8.
9.
10.
11.
12.
13.

A. Kumar, H. Dayanidhi and V. Kumar KS

Do you ever drive above the speed limit?
Have you ever cheated on your significant other?
Do you eat steak?
How do you like your steak prepared?
What is your Gender?
What is your Age?
What is your Household Income?
What is your Education?
What is your Location?

Personal data was anonymised and the data was published. The machine learning algorithm is used to predict ”what type of steak the individual likes”.The inversion attack tried to predict a sensitive data such as
“whether the person has cheated on his/her significant other”. The adversary had access to anonymised data, the model, the result, and other quasi-identifiers in the data set. Moving ahead, we even try figuring out what we say is the “most impactful attribute”. What we mean by “most impactful attribute” is that we remove data pertaining to a specific attribute (along with zero knowledge) and check what is the corresponding e↵ect on the accuracy of the inversion attack.

5
5.1

Inversion Attack - Results
Regression Tree

Decision tree learning uses a decision tree as a predictive model which maps observations about an item to conclusions about the item’s target value. Decision trees where the target variable can take continuous values (typically real numbers) are called regression trees [?]. In our initial analysis, we have used Regression
Trees to predict what type of steak the individual likes his steak prepared e.g Rare, Medium, Medium. Given the model, the result derived (type of steak the individual likes) and all the remaining attributes except
‘whether the person has cheated on his/her significant other”, the inversion attack tries to predict whether an individual has ever cheated on his significant other. The ability to predict such a sensitive attribute might be considered to be extremely privacy invasive.

Our analysis is not just restricted to the dataset for a given training size. Infact, we have analysed the accuracy of the inversion attack for the dataset using regression tree as a machine learning model for di↵erent sizes of the training and testing dataset as well. This would enable us to draw conclusions regarding the optimal size of the training set for future analysis.
Our analysis raised a new question: Do presence or absence of di↵erent attributes (or quasi identifiers) increase or decrease the accuracy with which a sensitive attribute can be accurately predicted/backtracked

Privacy Snooper: IOT

7

via inversion attack? Such an insight has immense repercussions for future analysis. Let us give an example to explain the idea:
Our experiments begin with us trying to predict a specific output for the attributes of a dataset, and the model being used given. This could be likened to a ML-as-a-service cloud system, where the user supplies the service with a dataset, is aware of the data model being used, and receives the desired output from the system. Now an adversary trying to figure out a sensitive attribute associated with the database might use the output released by the service, knowledge of the data model being used and other quasi identifiers in the database. For the purpose of this research, we have previously been assuming that the adversary is aware of all quasi-sensitive attributes except the sensitive attribute, surely along with the output of the database and data model used. However, let us assume that the adversary is now unaware of 2 attributes, the sensitive attribute and one of the quasi-sensitive identifiers. We shall now try to predict the sensitive attribute for this database given the data we posses. The accuracy with which this prediction can be made would give us what we call the ”weight” of the attribute, or the accuracy associated with the backtracking given the absence of a specific quasi-identifier. Qualitatively, what this tells us is the attribute that should not be released or known publicly at all costs. This would ensure that even though the adversary might try predicting a sensitive attribute in such a scenario, he wouldn’t be very e↵ective in it. A formal definition for the algorithm used to analyse this question is as follows:

Algorithm 2: Algorithm for Inversion Attack

1
2
3
4
5
6
7
8
9
10

11
12
13

Input: A Dataset with one or more sensitives
Result: A graph for accuracy of the attack for a given begin l=0; for Each Dataset do for Pick a fixed training and testing dataset split do for Each Classification based algorithm in set - classification tree, regression tree do
Train and acquire the model;
Pick a secondary missing attribute ↵; for Each ↵ do for Each row in dataset do
G given model and output value for a row, run inversion attack by ignoring the value of attribute on the model by substituting all possible values for the sensitive attribute and find its accuracy;
Find average accuracy of the inversion attack over all the rows;
Plot a graph for accuracy of the attack for a given;
Algorithm for computing accuracy for inversion attack given ignoring the value of attribute on the model; 8

5.2

A. Kumar, H. Dayanidhi and V. Kumar KS

Classification Tree

The Classification Tree is just like the regression tree, except the fact that the target variable can take a finite set of values [?]. Here, we have used Classification Trees to predict what type of steak the individual likes his steak prepared e.g Rare, Medium, Medium. Given the model, the result derived (type of steak the individual likes) and all the remaining attributes except ‘whether the person has cheated on his/her significant other”, the inversion attack tries to predict whether an individual has ever cheated on his significant other. This thereby presents us with a situation identical to what we had in hand in the previous subsection, except that we replicate everything for the case of a classification tree.

We have again extended our analysis of the most ”impactful attribute”

Privacy Snooper: IOT

6

9

Highlighted Observations and Conclusion

Our observations for the experiments we have run is multifaceted. We start with presenting our observations for accuracy of inversion attacks on Regression and Classification Tree:
1. We expect increase in e ciency of inversion attack as training data size increases for Regression Tree.
2. Regression tree classifier fits the pattern.
3. Classification tree classifier shows anomalous behaviour for small percentages of training data size (or
10 and 20 percent in the graph).
4. We suspect the behaviour is due to overfitting of data
5. Our method to measure accuracy of inversion attack = average of its accuracy for di↵erent rows. Overfitting might lead to a skewed distribution of accuracy with accuracy for training set data being very high. 6. Conclusion: Average Accuracy not a e↵ective measure of inversion attack e ciency
We now present our observations for accuracy of our inversion attack given the absence of specific quasisensitive identifiers.
1. Inversion Attack accuracy is the lowest in the absence of the attributes: 1. Household income 2. Location
( Census Region ) . This implies that if the data administrator can ensure that quasi-sensi tive identifiers like Household income and Location could either be removed or prevented from falling into the hands of the adversary, the accuracy with which he/she can predict a sensitive attribute like ”if a person has cheated on your significant other ” is decreased.
2. Inversion Attack accuracy is the highest in the absence of the attributes: 1. Education 2. Gender
3. Conclusions: To prevent inferences on whether a person has cheated on their significant other, it might be useful to suppress data on the attributes: Household income and Region to which the person belongs.

7

Future work

As described above, household income has a significant impact on the accuracy of the inversion attack.
While there might be many implications of this co-relation, we are interested in determining how this can be used to preserve the privacy of a user. In context of the data set used in the paper, we plan to run an experiment in which the initial machine learning algorithm is trained without the household income attribute, e↵ectively suppressing the said attribute from the training and test dataset. The purpose of this experiment is to determine how this new machine learning model compares in accuracy to the one which

10

A. Kumar, H. Dayanidhi and V. Kumar KS

was trained with the dataset that contained household income attribute. From this experiment we plan to determine the e↵ect a certain attribute has on the usability of the machine learning model, that is accuracy with which it can predict the kind of steak a certain participant prefers. If the accuracy of the new model varies only by a certain pre determined threshold, then we can e↵ectively remove that attribute from the entire dataset and use only the new machine learning model for prediction such that it would also be more privacy preserving.

Acknowledgements
We would like to sincerely thank Dr. Fredrikson for his guidance and sharing his prior expertise in tackling this problem . We are also highly thankful to Dr. Anupam Dutta and Mr. Amit Datta for their continuous feedback on our work.