KETL is different from Kettle, another open source ETL tool. You can read more about the similar names here at Nicholas Goodman’s blog. While Kettle is GUI oriented, KETL is scripted and probably more robust.

Read the KETL training doc to know more about its architecture and usage.

Note: To go through the examples here, you would need MySQL database and DB Designer for database modeling. You can either install DecisionStudio Professional (download from sourceforge), which has both MySQL and DB Designer along with many other analytics goodies, or else you can install them individually from MySQL website and DB Designer website. You would also need to download the sample foodmart database available along with DecisionStudio Professional.

Now let’s assume you are an IT person at FoodMart (a hypothetical retailer) who has decided to build a sales data mart as the first step in rolling out comprehensive analytics. In discussions with the sales department you have figured that the no. of units sold, dollar amount of sales, and the number of unique customers in a segment are the main metrics they look at. Digging deeper you figure that the sales guys are likely to want analysis by product, product category/class, brand, store location (city, state, region, country, …), customer demographics, and also by individual promotions and promotion categories. It may not be explicitly mentioned, but the metrics would also be analyzed by time (day, week, month, quarter, …)

Now that you have figured out the business metrics to be measured, this gives you the facts you would need in the data mart ‘fact table’ for calculating them. Similarly you have figured out the potential segments for analysis, and that gives you the ‘dimensions’ for analysis. The ‘fact table’ linked to the ‘dimension tables’ makes up the ’star-schema’ (because of the star-like structure), also known as the data mart.

With this information in place, we have the high level Dimensional Model for Sales.

Sales_Fact_1998 is the main fact table that has sales information by store/location, product, time, customer, and promotion. Correspondingly there are 5 dimension tables joined to the fact table through foreign keys in the star-schema.

The dimension tables in turn have detailed data that can now be used for defining ad-hoc analysis segments. For example, we can put demographic filters on the customer dimension (say age<30, married, college-educated), choose specific product class(es) in the product table (say Dairy Products), specify a limited time period, and then get our metrics calculated for the ad-hoc segment.

The image below shows the detailed information available in the dimension tables for defining ad-hoc segments.

You can download the data model here, and then open the saved model using DB Designer in DecisionStudio Professional (Start -> Program Files -> DecisionStudio Professional -> Data Analyst -> DB Designer Workbench). You can see other tables in the FoodMart database by scrolling around on the canvas (scroller in top-right corner).

Do note that our dimensional model for sales covers only a small relevant set of tables from the entire FoodMart database. You can load the entire downloaded FoodMart data into MySQL as outlined here, and can query on the data using Query Browser (Start -> Program Files -> DecisionStudio Professional -> Data Analyst -> MySQL Query Browser).

Open source BI helps SMBs by reducing the total cost of ownership by eliminating license costs. It provides greater flexibility as you can pick and use only what you want. This in turn helps keep implementations short and manageable.

Rick correctly points out that simplicity and focus on the essential is the key to promote adoption of open source BI. Adding unnecessary features and functions adds to the complexity and implementation costs. It is important for open source BI to keep this in mind.

Thanks Shane, for mailing me this link.

While R is used a lot for statistical research, its adoption for econometrics has been comparatively slower, possibly due to terminology differences.

An excellent text called “Econometrics in R” by Grant V. Farnsworth (PDF) provides a effective hands-on intro to the most common things you would find yourself doing including Time Series, regressions, Plotting, etc. Have a look.

DecisionStudio Professional (DSP) is an advanced graphical data mining, reporting, modeling, and analysis environment built on top of the best-of-breed open source projects. Some of these include:
— Optimized MySQL database as data warehouse platform
— SQL Workbench (MySQL Query Browser and DBDesigner) for Data Analysts
— R environment for statistical analysis and modeling
— iReport Reporting GUI and JasperReport reporting library
— Python with Boa Constructor IDE for application and GUI development

DecisionStudio Professional is the only end-to-end open source analytics platform that provides comprehensive capabilities to each role. Data Analysts get to store, process, and publish data on a standard MySQL platform; Reporting Analysts would like iReport and the integration with Office tools; and Modelers would love the excellent R Environment. It also includes Python along with a drag-n-drop GUI building environment for analytics Application Developers.

You can find out more about DecisionStudio Professional at decisionstudio.com, and can download your copy at Sourceforge.net. Click here to download the product brochure (PDF).

Go ahead, it’s completely free and will always stay so.

An Analytics group of yesteryear was usually made up of statisticians (and technologists) that would huddle together in their corners, engaged in a black magic which was ill understood by the business. Instead of breaking down information silos, Analytics shops were themselves becoming silos that worked on their own agendas and often had ill-concealed disdain for ‘business’. This has changed over time and an effective analytics group cannot afford to be disconnected from the business.

While the processes have matured in Analytics today, unfortunately the commercial tools have not, and still continue to make some invalid assumptions. Foremost among these is the assumption that all of Analytics is usually done by a single person (or a bunch of similar know-it-all-magicians).

This is not true.

If Analytics delivers Business Value (as everyone seems to be loudly proclaiming these days), there must be a value chain! And there must be some distinct roles and activities that lead to the creation of this value (in the form of better decisions that ultimately make more money for the business)!!

The Analytics Value Chain depicted above essentially shows the distinct roles that would exist in an Analytics group. There could be more roles, or a single individual might be handling multiple roles, but these are needed to ensure end-to-end delivery of comprehensive analytics. If you have been doing analytics, the diagram should be self-explanatory, otherwise I’ll elaborate upon it in the next post.

Did the pre-sales guy from that analytics vendor tell you that? Or did he push a product that can do it all but no one can fully comprehend and use?

For sanity’s sake, always keep the value chain in mind. If nothing else, it would mean you’d be clearer about what those twenty analysts are doing staring at those screens all day long.

In this post we would be setting up OLAP reporting for a hypothetical retailer called FoodMart that sells various grocery products in a chain of stores across US, Canada, and Mexico.

Assuming that you are running MS Windows on your machine, we would need:

JDK 1.4.2 or above downloadable from Sun’s java download page (I used j2sdk1.4.2)

Microsoft Access to act as the data store (drop a comment if you do not have MS-Access and I can provide detailed instructions on setting it up on another database such as MySQL)

Tomcat 5 from apache.org. Go to the bottom and download Windows Binary Distribution (I am using jakarta-tomcat-5.0.28.exe).

Mondrian along with FoodMart data from sourceforge.net

So now that you have downloaded all the software required, we can go about setting things up.

Install JDK (say in c:\j2sdk1.4.2). This may require you to reboot your machine a couple of times. Do not install Tomcat yet.

Right click on My Computer, select Properties -> Advanced -> Environment Variables. Create a new environment variable JAVA_HOME and point it to your JDK instalation (c:\j2sdk1.4.2)

Now install Tomcat (say in C:\Program Files\Apache Software Foundation\Tomcat 5.0). It should get installed as a service.

Go to Control Panel -> Administrative Tools -> Services to check if Tomcat5 service is running. If not, start the service.

Unzip Mondrian2.0.1.zip somewhere (say C:\Mondrian). Copy mondrian.war file from C:\Mondrian\mondrian-2.0.1\lib folder and place it in the webapps folder of your tomcat installation (C:\Program Files\Apache Software Foundation\Tomcat 5.0\webapps).

Within the unzipped Mondrian (in demo/access folder), you fill find a MS Access database file called MondrianFoodMart.mdb. This is where the data is going to get picked up from once we create a ODBC DSN.

Go to Control Panel -> Administrative Tools -> Data Sources (ODBC) -> System DSN. Create a new ODBC DSN called ModrianFoodMart pointing to the MondrianFoodMart.mdb above. Make sure you get this step correct (drop a comment if you need help).

Okay, now time to go see the brand new reports by pointing your browser to http://localhost:8080/mondrian/. If everything has gone right, you should see a page with a couple of links there. Click the first link called “JPivot pivot tables” under the heading “Mondrian Examples”. You will see a nice little report showing Unit Sales, Store Cost, and Store Sales for our FoodMart.

The user-interface is quite intuitive, so play around by clicking on various buttons. Do not forget to try out the top-left button (OLAP Navigator) that allows you to define your own reports by selecting measures, rows, columns and filters.

Have a go. And do let me know.

The OLAP Reporting revolves around two simple concepts: Dimensions, and Measures.

A Measure is any business metric that you want to calculate and report on. It can range from simple stuff like Dollar Sales (total sales revenue), Unit Sales (no of units for a particular product sold), Profits, Current Inventory Level, etc. to more complicated computed fields like Average Cost, Average Profit per Unit, % Margin, etc.

It should be noted that a Measure can have different values depending upon the context – whether you are looking at All Locations or a particular City, or whether you are looking at annual figures or weekly figures, or daily figures, whether you are looking at a particular demographic segment, etc. A measure’s value depends upon the specific subset of the entire database that you are looking at (hence the term “slice-and-dice” commonly used with “cubes”). The specific context for the calculation of Measures is provided by Dimensions.

Dimensions help subset the database to get to the specific information you are interested in. You may want to subset by Geography, Time, Product/Service Offering, Customer Demographics, etc. For example, I may want to analyze Unit Sales (Measure) for a particular product (specified on Product Dimension) in New York (specified on Geography Dimension) for the month of Jan 2006 (Time Dimension). I could then drill-down on Time Dimension to get the daily sales for the product.

OLAP tools provide this functionality through a user friendly GUI. The commercial OLAP systems are extremely expensive, but that would change with the advent of open-source OLAP tools.

An excellent open-source OLAP server is Mondrian that allows you to do OLAP cube reporting through a JSP web-based interface called JPivot.

The two together provide an excellent reporting system that can be quickly deployed on an existing datawarehouse to deliver impressive reports to end users.

In the next post we will set up a quick reporting system using Mondrian, JPivot, Tomcat and MySQL/MS-Access.

Now back to work.

A concern some of you have raised about the Consumer Finance data model we discussed (see Open Source Analytics in a month and subsequent posts) is that it appears far too simple to be able to deliver any analytical value to the business. Wouldn’t we be needing the behavioral, payment, response, clickstream, usage data, blah-this, blah-that, and blah-other in order to deliver any value? Isn’t a three table (Loan, Customer, and Marketing) demo too simplistic to be of any real use?

This post is really about answering this. Get out of the hype-victim mode and start thinking. If you look close enough there is enough you can do with just this much data. And in this post we explore just that.

Let us look at various categories of analytics we could do with this data.

1. Operational Sales Reports and MIS

2. Credit and Collections Reports

3. Predictive Modeling and Customer Segmentation

I just put this list down in 10 minutes. There’s a lot more that can be done, but this is just to give you a sense of what all is pssible even with seemingly limited data. You just need to look closer and put yourself in the other person’s shoes.

Okay, so now we have done away with your top two excuses for not doing analytics yet.

Any more excuses? Cool, we can now go ahead and start doing things.

Keep watching…

Today we shall design a Data Warehouse for the data.

In Consumer Finance Data Model – I, we described what AFS (our fictitious consumer finance organization) does, and the main pieces of information it deals with. To summarize, here are the pieces of data reproduced below:

Loan Level (specified for each loan):

LoanID, Product (auto/Personal/etc.), Amount Finance, Asset Cost, Loan Duration, Interest Rate, EMI Amount, Disbursal Date, and No of Installments. Also, POS (Principal Outstanding), DelPOS, Bucket, and Late Fee.

Customer Level (specified for each customer):

CustomerID, Name, Address, email, Age, Sex, Marital Status, Educational Qualification, Income, Years in Current Job, etc.

Marketing Information:

CustomerID, PromoID (uniquely identifies the prmotion or marketing offer), PromoDate, Product, and ConversionFlag (True if customer took up the offer within one month, False otherwise).

Listing down the data elements in this manner gives us a direct insight what the basic tables in the DW would be ad also their structure. We now know that we need at least thee tables, namely: Loan_Table, Customer_Table, and Promo_Table. And you guessed it right, we will be doing this on MySQL, the most widely used open source database.

Homework:
1) Download and install MySQL Server and MySQL Query Browser from theMySQL website.
2) Open MySQL Query Browser (connected to the just installed server) and create the three tables above. Query Browser is an excellent GUI tool and it shouldn’t be tough to find your way around.

If you are getting stuck, and/or need more specific help, please drop a comment below and I’ll help you out.

Keep walking…