An Empirical Analysis of the Value of Complete Information for eCRM Models Author(s): Balaji Padmanabhan, Zhiqiang Zheng and Steven O. Kimbrough Reviewed work(s): Source: MIS Quarterly, Vol. 30, No. 2 (Jun., 2006), pp. 247-267 Published by: Management Information Systems Research Center, University of Minnesota Stable URL: http://www.jstor.org/stable/25148730 . Accessed: 01/03/2013 09:24
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at . http://www.jstor.org/page/info/about/policies/terms.jsp

.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

.

Management Information Systems Research Center, University of Minnesota is collaborating with JSTOR to digitize, preserve and extend access to MIS Quarterly.

http://www.jstor.org

This content downloaded on Fri, 1 Mar 2013 09:24:20 AM All use subject to JSTOR Terms and Conditions

Padmanabhan

et al./The Value

of Complete

Information for eCRM

Models

Qjarteriy of the Value An Empirical Analysis Information for eCRM Models1
By: Balaji Padmanabhan Operations and Information Management Department The Wharton School University of Pennsylvania 3730 Walnut Street Philadelphia, PA 19104-6366
U.S.A.

of Complete

Abstract
Due to thevast amount of user data tracked online, theuse of data-based analytical methods is becoming increasingly com mon for e-businesses. Recently the termanalytical eCRM has been used to refer to theuse of such methods in theonline

balaji@wharton.upenn.edu Zhiqiang Zheng A. Gary Anderson School of Management University of California, Riverside Riverside, CA 92521
U.S.A. eric.zheng@ucr.edu

world. A characteristic ofmost of the current approaches in eCRMis that theyuse data collected about users' activities at a single site an only and, as we picture argue of user in this paper, activity. However, this can it is incomplete

present

possible across-site to obtain a complete picture of user activityfrom data on users. Such data is expensive, but can be

obtained by firms directlyfrom their users orfrom market data vendors. A critical question is whether such data is worth obtaining, an issue that littleprior research has addressed. In this paper,

Steven O. Kimbrough Operations and InformationManagement
Department

present an empirical analysis of themodeling benefits that can be obtained by having complete information.Our results suggest that the magnitudes of gains that can be obtained

using

a data

mining

approach,

we

The Wharton School University of Pennsylvania 3730 Walnut Street Philadelphia, PA 19104-6366
U.S.A.

from complete data rangefrom afew percentage points to 50 percent, depending on theproblem for which it is used and theperformance metrics considered. Qualitatively we find that variables related to customer

kimbrough@wharton.upenn.edu

sity are particularly importantand these variables are diffi cult to derive from data collected at a single site. More we importantly, find that a firm has to collect a reasonably

loyalty

and

browsing

inten

large amount of complete data before any benefits can be reaped and caution against acquiring too littledata.

Veda

Storey was the accepting senior editor for this paper. Chrysanthos Dellarocas, Maytal Saar-Tsechansky, and Young Rhu served as reviewers. The associate editor chose to remain anonymous.

Keywords: eCRM

Data mining, incomplete data, information value,

MIS

Quarterly

Vol. 30 No.

2, pp. 247-267/June

2006

247

This content downloaded on Fri, 1 Mar 2013 09:24:20 AM All use subject to JSTOR Terms and Conditions

Padmanabhan

et al./The Value

of Complete

Information for eCRM

Models

Introduction

M__________________________

(Swift 2000) has been used to refer to theuse of data-based analytical methods for customer analysis, customer inter actions, and profitabilitymanagement. A characteristic of most of the existing approaches to eCRM is that they build profiles and models based on data collected by a single Web site from users' interactionswith the site. In this paper we refer to such data as site-centric data, which we define to be clickstream data collected at a site augmented with user

designing effective CRM strategies (Padmanabhan and Tuzhilin 2003). This is particularly the case in the online world where Web servers automatically collect vast amounts of behavioral data on customers. The termanalytical eCRM

Effective customer relationshipmanagement (CRM) is impor tantfor any business (Pan and Lee 2003, Varian 2001). Ithas been shown that customer data can play a critical role in

In one case Reconsider the site-centric data for Expedia. (user 2), the first three pages result in the user making a purchase at thenext page. In the other case (user 1), the first threepages result in no purchase. Expedia sees the "same" initialbrowsing behavior, butwith opposite results. However sophisticated the analytical methods used, it is difficult to differentiate these two sessions based on site-centric data alone. Nevertheless, most conventional techniques implicitly tryto do so, coerced by incomplete information.

The example above suggests thatuser-centric data may be more informative than site-centric data for problems in eCRM. This observation is indeed actionable since tech nologies exist to collect user-centric data. Firms can obtain user-centric data in one of two ways.

1.

demographics and cookies (Sen et al. 1998). In this sense, traditional approaches are myopic, they are based on firms building models from data collected at their site only. However, the myopic nature ofmost currenteCRM methods is not due to the fact that site-centric data is adequate for understanding customer behavior; rather, it is due to the nature of data ownership: most sites only have access to their own log files. For example, consider two users who browse the Web for air tickets. Assume that the first user's session is as follows: Cheaptickets j, Cheaptickets 2, Travelocity]f Travelocity 2,
Expediaj, Expedia2, Travelocity3, Travelocity4, Expedia2,

Firms can get customers todownload client-side tracking software thatcaptures a user's online activities. Clearly in this case, firms also have to provide appropriate incentives forusers and should be explicit to theirusers about what data is tracked.

2.

Market data vendors, such as Netratings and comScore Networks, collect user-centric data by signing up panels and using client-side tracking software to capture all of the browsing activities of their panelists. Firms can purchase user-centric data directly from such vendors.

at Cheaptickets3 where Xt represents some page /, Website X and in this session assume that theuser purchases a ticket at Cheaptickets. Assume that the second user's session is purchases a ticket at Expedia (in thebooking page Expedia4, inparticular). Expedia's (site-centric) data would include the following: Userl: Expedia}, Expedia2, Expedia3 User2: Expediaj, Expedia2, Expedia3, Expedia4
Expedia2, Expedia3, Expedia4 and that this user

A firm's decision to acquire such data should be based on whether the benefits exceed the costs of obtaining the data. In thispaper, we focus on quantifying the gains that may be obtained fromuser-centric data, and on quantifyinghow much user-centric centric data. data By is necessary doing so, to obtain this paper provides to outperform answers site that

Expediaj,

will help firms in understanding the benefits thatmay be obtained from having complete information. Actual deci sions, of course, will also depend on the particulars of the individual businesses. Specifically, we consider three important and common problems in eCRM and for each problem we evaluate how user-centric models

We define user-centric data to be site-centric data/?/?s data on where else the user went in the current session. In this sense, user-centric data

of these threeproblems, we answer the following questions: 1. What from compare

to site-centric

models.

For

each

centric data. In the above example, theuser centric data for Expedia is: Userl: Cheaptickets lf Cheaptickets 2, Travelocity]t
Travelocity2, Expediaj, Expedia2, Travelocity3,

is the "complete"

version

of

site

is the magnitude of the gain that can be expected acquiring complete user-centric data, and what

factors drive the gains that are obtained? This is an important question since the answers to this can help understand the benefits of user-centric data. However, alone

Travelocity4, Expedia3, Cheaptickets3 User2: Expedia]f Expedia2, Expedia3, Expedia4

froma practical perspective, although there may be gains from user-centric data, cost considerations may

prevent firms frombeing able to acquire user-centric data

248

2006 Vol. 30 No. 2/June MIS Quarterly

This content downloaded on Fri, 1 Mar 2013 09:24:20 AM All use subject to JSTOR Terms and Conditions

Padmanabhan

et al./The Value

of Complete

Information for eCRM

Models

from all their customers. we study. question that 2.

This motivates

the second

choices for these three dimensions, and based on this discussion we identify the specific values that need to be estimated. If it is only possible to acquire user-centric data for a sample of customers, how much data should be acquired inorder todo better than with site-centricdata alone? We use the term critical mass to refer to this number. In addition to thepractical reason for studying thisquestion, this is interesting from a research perspective since it addresses a tradeoff between having more (less informa site-centric data or less (more informative) user tive) centric data.

Choice of theProblems
In order to compare models built on site-centric and user centric data, we consider the following importantand com mon problems in eCRM: Predicting repeat visits of users. This is important in the context of understanding loyalty and switching propen sities of online users. In particular,we build models that, at the end of each user session, predict iftheuser is likely to visit a site again at any time in the future. In prior

The main results are that themagnitudes of the gains are significant,but the actual values depend on theperformance metric. User-centric data can increase

hard, if not impossible, to determine accurately from site centric data. Further, the criticalmass results reveal that the criticalmass numbers are fairlyhigh, ranging from 25 to 45 percent depending on the problem and metric considered. This is an important result since it suggests that if firms acquire partial user-centric data, then acquiring too littlecan such data can actually be worse than site-centric models.

of thedesired targetvariable by 10 to 20 percent for the three problems considered. The gains are more striking for com puting lift gains, where the magnitude of thegains isbetween 20 and 50 percent. Further, analyses of themodels built reveal several importantpredictors. In particular, the key predictors fromuser-centric data capture effects thatrelate to customer loyalty and browsing intensity?both ofwhich are

the predictive

accuracy

work, Lee (2000) modeled repeat visit behavior of online users using an NBD (negative binomial distribution) model. Chen and Hitt (2002) studied consumers' switching behavior between online brokers based on repeat visits using a logitmodel.

Predicting likelihood of purchase at a site. Within this category we identify two subproblems. The first is a real-time be counter-productive in thatuser-centric models built from

at every successive click, the task is topredict if thisuser is likely to purchase within the remainder of the session. We termthis the within-session prediction problem. The second is a prediction problem with a longer time horizon: at the end of each user session, predicting if this user is likely to purchase during any subsequent session in the future, the future-session prediction problem. In prior work, VanderMeer et al. (2000) developed a personalization system to predict a user's next access in

prediction

task: within

a current

user's

session,

The rest of this paper is organized as follows. In the next section, we explain our choice of the eCRM problems con sidered, themodeling technique used, and the performance metrics of the models (for which the gains are studied). Based on thisdiscussion we state the goals of thepaper more specifically in terms of determining two 3>. such that addjyage^p^,...^^) returns , S{ and Cj is a set of tuples of the form
IQ if page in is the jthpage accessed is .
Also we define function fragmentiS^

define function kth-click(Sx, j) that returns a tuple , S{ as determined from the time each page is accessed. For instance, in the above example

kth-click(Sl9

= m) -1), |Cj|),where Sj . fragment^, j, fraglength)is the set of all tupleskth-click(Sx, such that j ? m ? minimum( (j + fraglength = For examplc/ragw^S!, 02/01/200123:45:15), (IP, 128.122.195.3)} >, N){i

}

output'Dp'

=

1}

| FigureA1. AlgorithmProbabilistic Clipping

Appendix

B

Within-Session Prediction IHHIIH^^B^IIIHII^HBHHHiH^HHIHi^HBHIHI Metrics for
Table we use the same table to explain themetrics For space considerations, for the within-session prediction problem. 1 through 12 in 12 site-centric ones (numbers include 30 independent variables: Both problems and repeat-visit problems. are different: ones (numbers for these two problems the dependent variables 16 through 33). Also, user-centric the table) and 18 additional for repeat-visit in the future (after the session); indicates a user will book the dependent variable if prediction, (2) (1) for future-session share the same metrics that even though the two problems Note it indicates ifhe will repeat visit in the future (after this session). prediction, section of the paper. in the "Methodology" as within-session the values of metrics are generated from a different dataset as detailed prediction, B1 details themetrics for future-session

2006 Vol. 30 No. 2/June MIS Quarterly

265

This content downloaded on Fri, 1 Mar 2013 09:24:20 AM All use subject to JSTOR Terms and Conditions

Padmanabhan

et al./The Value

of Complete

Information for eCRM

Models

Table

B1.

Metrics

for the Within-Session

Prediction

Problem_

No._Variable_Description_ 1_gender_"1"?Male, 2_age_Age 3_income_Income 4_edu_"0" of the "0"

Female_

user_ of the user_ or high school

less,

"1"-- college,

"2"

-

post

5_hhsize_Size
6_child_"1"

of house have, "0"

college_

hold_ not have_

7_booklh_No.
_8_sesslh_No.

of bookings the user made at thissite inthe of sessions to this site so far_

past_

measure of the frequency of purchases to thissite, defined as booklh/sesslh_ spent in thissite so far in 10_minutelh_Time minutes_ Average hits per session to thissite_ 11_hpsesslh Average timespent per sessions to thissite_ 12_mpsesslh of hits to this site up to this point inthis 13_hitlc_No. session_ spent up to this point inthissession_ 14_minutelc_Time ifthissession occurs on 15_weekend_Indicating weekend_ of past bookings of all sites so 16_bookgh_No. far_ 9_freqlh_A
17_sespsite_Average sessions per site so

18_sessgh_Total 19_freqgh_A 20_minutegh 21_hpsessgh 22_mpsessgh 23_awareset
24_basket_Average 25_single_Percentage

far_ measure of the frequency of purchases across all sites, defined as bookgh/sessgh_ Total minutes of all sites_ Average hits per session_ Average minute per session_ Total no. of unique shopping sites visited_ no. of shopping of single-site sites visited per session_ sessions_

no. of sessions visited of all sites so

far_

26_booksh_Percentage 27_hitsh_Percentage 28_sessh_Percentage 29_minutesh 30_entrate_No. 31_peakrate_No.

of total bookings are to this of total hits are to this site_ of total sessions are to this

site_

site_ the user spend themost time within thissite/total sessions_ of sessions end with this site/totalsessions of this 32_exitrate_No. site_ of sessions startwith thissite/totalsessions of this 33_SErate_No. site_ hits of all sites inthe current 34_hitgc_Total session_ No. of shopping sites inthis 35_basketgc session_ Time spent of all sites inthis 36_minutegc session_ if thissession uses search 37_SEgc_Indicating engines_ this site isan if 38_path_Indicating entry/peak_ to this site/ hits to all sites inthis 39_hitshc_Hits session_ Minutes to thissite/total minutes inthis 40_minutshc session_ 41 bookfut Binary dependent variable, indicating ifthis user isgoing to book inthe remainder of the session (afterthe clipping point) of sessions
Note: Variables 1 through 15 are site-centric variables; 16 through 40 are additional user-centric variables; variable 41 is the dependent variable.

site_ Percentage of totalminutes are to thissite_ of sessions startwith thissite/totalsessions of this

266

MIS Quarterly Vol. 30 No. 2/June 2006

This content downloaded on Fri, 1 Mar 2013 09:24:20 AM All use subject to JSTOR Terms and Conditions

Padmanabhan

et al./The Value

of Complete

Information for eCRM

Models

Appendix

C
M

Rationale Behind Variable Construction ^
The user-centric and the URL of variables

to us consisted of household ID (HHID), data provided the time a user visits each page, the duration of the visit, the site visited a set of the specific page. Each site is also pre-categorized into categories such as search engine, travel, etc. In order to develop can be identified. A central concept in the framework from the raw usage data, we used a framework within which these variables

is that of a session, which is a collection of hits (or clicks) by a single user during a single browsing period.8 For each session, the data is of the format (page1? timel5 domain^, Given this structure, for the current session we created features with respect to hit (pagek, timek, domain^. time, and site (domain). As Table C1 shows, features related to hits and time are identically created for both site-centric and user-centric (page), datasets. domains. With Hence sessions consist of pages from different respect to domain, all site-centric sessions are under the same domain, while user-centric for user-centric features that do not have an equivalent in site-centric data: datasets, we construct three types of cross-domain features that capture and ends. tells us behavior. the time sequence of visits to different sites. Three important characteristics of a path are where the

(1)

Path-related session

starts, peaks, data also

(2)

User-centric mative about

information

about what

shopping sites is again relevant since for user-centric data. data also has

The use of search indicates

it broadly

is visiting. We note that two types of sites may be particularly infor is important since it suggests a focused need for the user. Visiting engines shopping interest in things that can be bought. Hence under site we identified two such variables

sites a user

(3)

User-centric

related market

share

information.

Specifically,

we

construct

time and hit related market

share metrics.

minutelh,

for the historical sessions, we also have information on hit, time, and site. From the site-centric data, we create four variables: sesslh, to this site, the From user-centric data, in addition, we know the share of a user's activities (see Table CI). hpsesslh, and mpsesslh of the past sessions. Thus we created three metrics and hitsh) describing the share of acti path, and the site characteristics (sessh, minutesh, and exitrate) describing the path, and five metrics (SErate, basket, single, awareset, and sespsite) vities; another three metrics (entrate, peakrate, Similarly, are important indicators of future purchases. the characteristics From site-centric data, we have of sites. Finally, past purchases describing to this site) and freqlh booklh (total number of purchases inwhich a booking was made at this site). From user of user sessions (percentage across sites), freqgh centric data, we know bookgh (total number of purchases at of user sessions inwhich a booking was made (percentage site), and framework. any booksh (the share of purchases to this site). Table CI details the site-centric and user-centric metrics developed under the

Table

C1.

Identified Site-Centric On Statistics

and User-Centric

Metrics

Under

the Framework

Granularity Level

User-Centric Site-Centric

Hit Hitgc ~ Minutegc, weekend Path Path
Time
-?-:?

Title
Miutelc, weekend

Current

^

,^

Session

. Sites Share Hit Sessions Time
Path

basketgc, SEgc ?;?-? minuteshc, hitshc Hpsessgh hpsesslh sesslh Sessgh Minutegh,, mpsessgh entrate, peakrate, exitrate

Minutelh,

mpsesslh

Share
Sites

sessh, miutesh, hitsh
SErate, single, basket, awareset, sespsite

Shopping User Demographics

bookgh, freqgh,booksh 6 variables 6 variables

Booklh,

freqlh

A common rule of thumb is to observe

the time associated with each hit and group consecutive

hits that are within 30 minutes of each other into a session

(http://w3 .org/WCA/1999/01 /Terms.html).

MIS Quarterly Vol. 30 No. 2/June 2006

267

This content downloaded on Fri, 1 Mar 2013 09:24:20 AM All use subject to JSTOR Terms and Conditions