[FRIAM] Subtle problem with BI

Sat Mar 3 15:58:34 EST 2007

These are issues I (and many others) have grappled with for many years. I 
have strong opinions that deftly straddle both sides. So - I can't be wrong! 
To address the points Mikhail raised, I'll use the context of using data to 
predict sales.

1) "we assume that our data reflect adequately business issues (customer 
behavior) "

The question here is what is "adequately", and what is "customer behavior". 
Defining these precisely is very important to developing an accurate, useful 
prediction system. Understanding what is "adequate" is tough. For the 
client, it initially means "better than what I do now." Later, it evolves 
into something like "Error = 5 - 10 %."  For sales prediction, this is an 
impossible standard. So in the end, the client will be unhappy!

Several problems cause this. First, the customers are not homogeneous. 
Different groups behave differently to the same stimuli. And the groupings 
you can develop of similarly behaving customers for one product is not the 
same as for another. I.e., knowing how a customer responds to a Coke 
promotion doesn't necessarily tell you how he/she will respond to a Tide 
promotion. Second, you don't always have the most important data you need. 
Normally for sales, you will have price and volume data for the item of 
interest and competitors (identifying competitors is another problem ...). 
But many important data pieces that have major effects on sales (or stock 
prices, inventory levels, etc.) are not what I call "observable" in the data 
the client can give you. This "unobservable" data can include a major sale 
on the item by the WalMart across the street from a store, a major snowstorm 
that keeps people out of the stores, errors in the shelf price tag, 
stockouts in the distribution chain, local population changes due to 
holidays, etc. While sometimes this "unobservable" data can be gotten, it 
takes a lot of work and is very expensive. Third, even though you may have 
what you think is lots of data (typical retail data sets hold tens of 
billions of transactions), it isn't enough! By the time you develop a model 
you think has all the important variables/features (e.g., price, time of 
day, day of week, day of month, month of year, prices of major competitive 
items in store, etc.), and develop a reasonable number of values for each 
that lead to different behavior, you find you have a very large 
multidimensional matrix, which for many of the elements will have only a few 
(0 - 10) observations. Theoretically, you need 20+ observations per element 
to give you statistically valid results. Fourth, often the data you get is 
"dirty", with e.g. price errors, unidentified replacement products, and so 
on. We have found that anywhere from 30 - 80% of the time required to do an 
analysis/model development task is needed to understand and clean the data 
the client provides.

There are of course other problems, but the ones above tend to be the most 
significant.

2) "we update (patch) our data-collecting software very often."

I don't understand why this is a problem. Normally, data collection software 
for business (e.g., Point of Sale cash register data) is pretty robust. I 
assume here he means that as new types of data (e.g., new 
variables/features) are discovered or developed and as dirty data is 
cleaned, that the models you develop will change. This should be done. The 
process we use to develop statistical BI models is a) clean the data, b) 
examine it to understand it as much as possible and identify important 
features/variables, c) talk to experts to develop "domain knowledge", d) 
develop with the client desired performance specifications, e) develop and 
test a model, f) figure out why the results are so bad, g) modify 
algorithms, add or subtract data types, h) repeat until results are "good 
enough", money runs out, client gets antsy, etc.

I think that changing your data structures and models is usually an 
important and necessary part of developing a model that will meet your 
client's accuracy requirements.

Nuff said.

Jack Stafurik

>
> Message: 1
> Date: Sat, 03 Mar 2007 11:23:20 -0500
> From: "Phil Henshaw" <sy at synapse9.com>
> Subject: Re: [FRIAM] Subtle problem with BI
> To: "'The Friday Morning Applied Complexity Coffee Group'"
> <friam at redfish.com>
> Message-ID: <000a01c75db0$426d38f0$2f01a8c0 at SavyII>
> Content-Type: text/plain; charset="iso-8859-1"
>
> I don't quite understand the details, but sounds link a kind of 'ah ha'
> observation of both natural systems in operation and the self-reference
> dilemma of theory.   My rule is try to never change the definition of
> your measures.  It's sort of like maintaining software compatibility.
> if you arbitrarily change the structure of the data you collect you
> can't compare old and new system structures they reflect nor how your
> old and new questions relate to each other.   It's such a huge
> temptation to change your measures to fit your constantly evolving
> questions, but basically..., don't do it.  :)
>
>
>
> Phil Henshaw                       ????.?? ? `?.????
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 680 Ft. Washington Ave
> NY NY 10040
> tel: 212-795-4844
> e-mail: pfh at synapse9.com
> explorations: www.synapse9.com <http://www.synapse9.com/>
>
> -----Original Message-----
> From: friam-bounces at redfish.com [mailto:friam-bounces at redfish.com] On
> Behalf Of Mikhail Gorelkin
> Sent: Tuesday, February 27, 2007 5:06 PM
> To: FRIAM
> Subject: [FRIAM] Subtle problem with BI
>
>
>
> Hello all,
>
>
>
> It seems there is a subtle problem with BI (data mining, data
> visualization, etc.). Usually we assume that our data reflect adequately
> business issues (customer behavior), and in the same time we update
> (patch) our data-collecting software very often, which reflects the very
> fact of its (more or less) inadequacy! So, our data also have such
> inadequacy! but we never try to estimate it 1) to improve our software;
> 2) to make our business decision more accurate. It looks like both our
> data-collecting software and BI are linked together forming a business
> (and cybernetic!) model.
>
>
>
> Any comments?
>
>
>
> Mikhail
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: 
> http://redfish.com/pipermail/friam_redfish.com/attachments/20070303/eb14ee4a/attachment-0001.html
>
> ------------------------------
>
> _______________________________________________
> Friam mailing list
> Friam at redfish.com
> http://redfish.com/mailman/listinfo/friam_redfish.com
>
>
> End of Friam Digest, Vol 45, Issue 3
> ************************************
>
>
> -- 
> No virus found in this incoming message.
> Checked by AVG Free Edition.
> Version: 7.5.446 / Virus Database: 268.18.6/709 - Release Date: 3/3/2007 
> 8:12 AM
>
> 