Data Mining, Analytics and BI roll-outs are unlike any other project your organization may have undertaken. Political and non-technical issues can derail the fragile project which is anyway struggling to handle ambiguous and constantly changing requirements.

Mark Demarest, in his classic 1997 article The Politics of Data Warehousing says:

Data warehousing projects are frequently side-tracked or derailed completely by non-technical factors, in particular the political treaty lines within the firm, and the politicized nature of data itself. Because data warehouses are infrastructure for sociotechnical systems (STSs) within the firm, politics and the exercise of power are inherent in data warehousing projects, and data warehouse designers have to adopt work practices and methods from non-technical disciplines, think of themselves in new ways, and employ some fairly sophisticated qualitatively sociological methods in order to optimize the chances for successful deployment of data warehouses.

The Politics of Data Warehousing also lists down 10 warning signals to detect politicization of the project, and 10 countermeasures – essentially the recipe for delivering on the project promise.

Larry Greenfield, in a more recent article titled Data Warehousing Political Issues identifies three common threads as:

1) Data warehousing imposes new obligations whose responsibilities are unclear
2) Data warehousing requires changes in processes that an organization is comfortable with
3) Data warehousing requires agreement on some, but not all, definitions of data.

Larry classifies the political issues into those that are within the IS organization (IS to IS), those that are between IS and the users (IS to Users), and those that are between users (User to User). Click here to read Data Warehousing Political Issues.

And don’t miss the article Data Warehousing Gotchas, listing some little known nuggets of wisdom and experience that can save your project. As Larry says, “Forewarned is forearmed!” Click here to read Data Warehousing Gotchas

SELECT table1.* FROM table1 LEFT JOIN table2 ON table1.id=table2.id WHERE table2.id IS NULL

Now suppose you want to search not from the entire table2, but from a subset of table2 (say CityID = 1). How do you do that? You can try and see for yourself that the following SQL is not the solution:

SELECT table1.* FROM table1 LEFT JOIN table2 ON table1.id=table2.id
WHERE table2.id IS NULL
and table2.CityID = 1

Note that this is because we are looking for records that do not exist in table2 and hence cannot have a where clause on table2 data.

Read the manual page a little further, and another user comment points out:

Conditions for the “right table” go in the ON clause.
Conditions for the “left table” go in the WHERE clause,
except for the joining conditions themselves.

That makes the solution as:

SELECT table1.* FROM table1 LEFT JOIN table2 ON (table1.id=table2.id and table2.CityID = 1)
WHERE table2.id IS NULL

Happy digging!

KETL is different from Kettle, another open source ETL tool. You can read more about the similar names here at Nicholas Goodman’s blog. While Kettle is GUI oriented, KETL is scripted and probably more robust.

Read the KETL training doc to know more about its architecture and usage.

Agile development is an approach that “cycles” through the development phases, from gathering requirements to delivering functionality into a working release. Two widely used agile development frameworks are the Microsoft Solutions Framework (MSF) and Extreme Programming (XP).

Agile development methodologies begin with ‘user stories’ which are brief 2-3 sentence problem statement. They tend to describe just enough of user requirements to get a rough idea of effort and scope. These are then used to develop multi-release plans that establish the roadmap for the team and establish the longer term vision for the team.

The actual development is done in frequent cycles of small releases (both fresh development releases as well stabilization releases). Stories are assigned to individual development cycles depending upon resources, time and requirements. This Resource-Time-Requirement triangle is actively managed through triage to make sure that release schedules are met.

Click here to read about applying Agile development methodologies to Data Warehousing projects.

Note: To go through the examples here, you would need MySQL database and DB Designer for database modeling. You can either install DecisionStudio Professional (download from sourceforge), which has both MySQL and DB Designer along with many other analytics goodies, or else you can install them individually from MySQL website and DB Designer website. You would also need to download the sample foodmart database available along with DecisionStudio Professional.

Now let’s assume you are an IT person at FoodMart (a hypothetical retailer) who has decided to build a sales data mart as the first step in rolling out comprehensive analytics. In discussions with the sales department you have figured that the no. of units sold, dollar amount of sales, and the number of unique customers in a segment are the main metrics they look at. Digging deeper you figure that the sales guys are likely to want analysis by product, product category/class, brand, store location (city, state, region, country, …), customer demographics, and also by individual promotions and promotion categories. It may not be explicitly mentioned, but the metrics would also be analyzed by time (day, week, month, quarter, …)

Now that you have figured out the business metrics to be measured, this gives you the facts you would need in the data mart ‘fact table’ for calculating them. Similarly you have figured out the potential segments for analysis, and that gives you the ‘dimensions’ for analysis. The ‘fact table’ linked to the ‘dimension tables’ makes up the ’star-schema’ (because of the star-like structure), also known as the data mart.

With this information in place, we have the high level Dimensional Model for Sales.

Sales_Fact_1998 is the main fact table that has sales information by store/location, product, time, customer, and promotion. Correspondingly there are 5 dimension tables joined to the fact table through foreign keys in the star-schema.

The dimension tables in turn have detailed data that can now be used for defining ad-hoc analysis segments. For example, we can put demographic filters on the customer dimension (say age<30, married, college-educated), choose specific product class(es) in the product table (say Dairy Products), specify a limited time period, and then get our metrics calculated for the ad-hoc segment.

The image below shows the detailed information available in the dimension tables for defining ad-hoc segments.

You can download the data model here, and then open the saved model using DB Designer in DecisionStudio Professional (Start -> Program Files -> DecisionStudio Professional -> Data Analyst -> DB Designer Workbench). You can see other tables in the FoodMart database by scrolling around on the canvas (scroller in top-right corner).

Do note that our dimensional model for sales covers only a small relevant set of tables from the entire FoodMart database. You can load the entire downloaded FoodMart data into MySQL as outlined here, and can query on the data using Query Browser (Start -> Program Files -> DecisionStudio Professional -> Data Analyst -> MySQL Query Browser).

Margaret Dunham, the author of Data Mining: Introductory and Advanced Topics once said:

Data mining research and practice is in a state similar to that of databases in the 1960s.

That is true, but will have to change. Databases have long been standardized, and data-mining & BI would get standardized in due course. Till then, I guess, we’ll have to do with whatever information is available, ignore the hype, and take the plunge for defining our own solutions.

So what does it take to deliver upon the BI promise? Data, for sure. And then you need a lot of meta data as well to make sense out of the data. And most importantly, the context for analysis – the what, why, and how of the effort. In today’s implementations, the context is implicitly identified and built into the solution, which may not be flexible enough to meet the changing needs of the users.

As an excellent effort in the right direction, Dratz recently published a white-paper outlining why BI should be viewed as an Abstraction Architecture for Information, and how it should be structured. And why a flexible BI architecure should have be the ability to apply dynamic contexts at run-time for the conversion of data into meaning. He proposes a very interesting piece therein: a Transaction Clearinghouse that would house the data till all pieces of an enterprise transaction (possibly spanning across multiple systems) are available, and would apply contexts dynamically to create meaning out of the data.

Even if you aren’t sure what all this means, do read Introduction to an Abstraction Architecture for BI white paper by Dratz. It will give you a very good overview of what BI is, why it is needed, and how the vision could be actualized.

In case you’d like to first read a little bit of background about the white paper, you can read his post Abstraction Architectures…or, Why Everyone Should be thinking of BI.

Do let me know what you think?

In this post, we’ll try to get some sanity around the concepts, without getting drawn (hopefully) into the crossfire.

A Data Mart is a specific, subject oriented, repository of data designed to answer specific questions for a specific set of users. So an organization could have multiple data marts serving the needs of marketing, sales, operations, collections, etc. A data mart usually is organized as one dimensional model as a star-schema (OLAP cube) made of a fact table and multiple dimension tables.

In contrast, a Data Warehouse (DW) is a single organizational repository of enterprise wide data across many or all subject areas. The Data Warehouse is the authoritative repository of all the fact and dimension data (that is also available in the data marts) at an atomic level.

Unfortunately, this is where consensus begins to break down into chaos. There are two broad schools of thought lead by Kimball and Inmon that disagree on the details.

Kimball School:
Ralph Kimball began with the Data Mart as a dimensional model for departmental data and viewed the Data Warehouse as the enterprise wide collection of Data Marts. This is the bottom-up approach. You may begin with the Sales Data Mart, after sometime you put in place the Ops Data Mart, and so on an so forth. If you want you could have even more specific Data Marts serving specific questions like customer Churn. If you take care of consistency of metadata (making sure each departmental Data Mart calls an Apple an Apple) and connectivity, you have a Data Warehouse. So the Data Warehouse is really a virtual collection of Data Marts collected together on a Data Warehouse Bus, and in that sense the data flows from multiple Marts into the Warehouse.

Inmon School:
Inmon’s approach is the exact opposite and avoids the problem of metadata consistency by looking at the Enterprise Data Warehouse as a single repository that feeds subject oriented Data Marts. You still have your Sales, Marketing, Ops and Churn Data Marts containing atomic or aggregated information, but they are based on the Data Warehouse and are really subsets of the data contained therein. This is the top-down approach.

Kimball’s approach is easier to implement as you are dealing with smaller subject areas to begin with, but the end result often has meta data inconsistencies and can be a nightmare to integrate. Inmon’s approach, on the other hand does not defer the integration and consistency issues, but takes far longer to implement (which makes it easier for the project to fail). Also, in my experience, organizations that are just starting to do analytics usually do not have the patience or commitment required for Inmon’s approach.

Any BI initiative is extremely iterative in nature. Unless you are confident that you would still have the CEO’s buy-in and a budget one year down the line, it might be better to begin with a Data Mart (to start delivering, and to manage expectations) keeping the meta data consistency requirements in mind, and then scale towards the Data Warehouse.

If you are interested in knowing more about the Great Debate, you can read an article by Katherine Drewek called “Data Warehousing: Similarities and Differences of Inmon and Kimball”.

P.S.: This is what I think, and I’ll be glad to hear of your experiences and views, specially if you have been a part of an Inmon style implementation.

DecisionStudio Professional (DSP) is an advanced graphical data mining, reporting, modeling, and analysis environment built on top of the best-of-breed open source projects. Some of these include:
— Optimized MySQL database as data warehouse platform
— SQL Workbench (MySQL Query Browser and DBDesigner) for Data Analysts
— R environment for statistical analysis and modeling
— iReport Reporting GUI and JasperReport reporting library
— Python with Boa Constructor IDE for application and GUI development

DecisionStudio Professional is the only end-to-end open source analytics platform that provides comprehensive capabilities to each role. Data Analysts get to store, process, and publish data on a standard MySQL platform; Reporting Analysts would like iReport and the integration with Office tools; and Modelers would love the excellent R Environment. It also includes Python along with a drag-n-drop GUI building environment for analytics Application Developers.

You can find out more about DecisionStudio Professional at decisionstudio.com, and can download your copy at Sourceforge.net. Click here to download the product brochure (PDF).

Go ahead, it’s completely free and will always stay so.

The Data Warehouse is the foundation of any analytics initiative. You take data from various data sources in the organization, clean and pre-process it to fit business needs, and then load it into the data warehouse for everyone to use. This process is called ETL which stands for ‘Extract, transform, and load’.

While the extraction (from files and/or databases) and load (into the DW) are pretty straightforward, the real juice of ETL is in the data transformations where the data is converted into a more business usable format. Typical Transformation tasks involve:
* Choosing the specific columns out of the available data
* Relating data from multiple sources
* Translating and/or encoding values (e.g. storing a CityID in the DW instead of the names to conserve storage and ensure integrity)
* Encoding free-form values from legacy systems (there can be a lot of valuable information in some of those memo fields)
* Deriving a new calculated values and Summarizing multiple rows of data

You can either write a specific processor (Python is a good choice as it aids rapid prototyping, is quite readable, and is easier to maintain) or can use an ETL Tool, a software application designed to aid the process. While ETL tools provide ease of use and rapid deployment, they often assume that the source data is cleaner, better organized, and less quirkier than it usually is. If there are a lot of quirks in the data, you might be better off writing one on your own. Some ETL tools provide a plugin architecture to handle this better.

Some of the ETL tools from the open source domain are listed below:
1) Kettle
2) Octopus
3) Clover ETL

And a good blog post listing numerous Java based options is http://www.manageability.org/blog/stuff/open-source-etl/view.

Please do let me know if I have missed some significant ones out. If you have used any of these, we’ll all appreciate if you can comment upon your experiences.

To design a good Data Warehouse that actually answers user queries effectively you need to do what is known as Dimensional Modeling.

In a DW, the data is kept in a standard, intuitive framework or structure that facilitates high performance access. This usually includes a Fact Table containing aggregatable facts (such as sum of sales amount) linked to one or more smaller Dimension Tables containing descriptive information (like time, region, product, etc. that you may want to analyze the facts by) in a star like structure called a Star Schema. This is exactly how your application database is not structured.

The process of identifying the facts and dimensions for a business scenario and designing the corresponding Data Warehouse is called Dimensional Modeling. Ralph Kimball’s The Data Warehouse Toolkit: The Complete Guide to Dimensional Modelingis an excellent text that is frequently referred to by amateurs and professionals alike. It is about how to actually design and build a repository that will deliver real value to real people. (If you still don’t have this book yet, please stop wasting your time and go get your copy now).

And while you go get that book, do read this article, an original classic by Ralph Kimball (from way back in August 1997) called the Dimensional Modeling Manifesto.
This is the article that sets the stage (and the manifesto) for the ideas developed by Ralph in his book. And like good wine, its relevance has only increased with time.

Read it, and then go get that book.