Top Resources for Analysts: Excel, Data Analysis, and Business Intelligence is another list you shouldn’t miss.

If you work with data for a living, the following sites are worth a visit (or a subscription) to learn from some of the best, most passionate practitioners.

Data Mining, Analytics and BI roll-outs are unlike any other project your organization may have undertaken. Political and non-technical issues can derail the fragile project which is anyway struggling to handle ambiguous and constantly changing requirements.

Mark Demarest, in his classic 1997 article The Politics of Data Warehousing says:

Data warehousing projects are frequently side-tracked or derailed completely by non-technical factors, in particular the political treaty lines within the firm, and the politicized nature of data itself. Because data warehouses are infrastructure for sociotechnical systems (STSs) within the firm, politics and the exercise of power are inherent in data warehousing projects, and data warehouse designers have to adopt work practices and methods from non-technical disciplines, think of themselves in new ways, and employ some fairly sophisticated qualitatively sociological methods in order to optimize the chances for successful deployment of data warehouses.

The Politics of Data Warehousing also lists down 10 warning signals to detect politicization of the project, and 10 countermeasures – essentially the recipe for delivering on the project promise.

Larry Greenfield, in a more recent article titled Data Warehousing Political Issues identifies three common threads as:

1) Data warehousing imposes new obligations whose responsibilities are unclear
2) Data warehousing requires changes in processes that an organization is comfortable with
3) Data warehousing requires agreement on some, but not all, definitions of data.

Larry classifies the political issues into those that are within the IS organization (IS to IS), those that are between IS and the users (IS to Users), and those that are between users (User to User). Click here to read Data Warehousing Political Issues.

And don’t miss the article Data Warehousing Gotchas, listing some little known nuggets of wisdom and experience that can save your project. As Larry says, “Forewarned is forearmed!” Click here to read Data Warehousing Gotchas

Technologists think, “Build it, and they will come.” But they’re building plenty of cool stuff, and consumers aren’t coming.

Techies are often so taken and smitten by their own technology that they fail to understand why and how it may not sell. Just because a technolgy is logically coherent and technically brilliant doesn’t mean it would sell. And sale is where the first ‘potential for value’ gets created, whether or not a technologist likes to admit it.

Most people will not switch to something new unless the perceived benefits far outweigh the perceived pain in switching. Pip Coburn, the author of “The Change Function: Why Some Technologies Take Off and Others Crash and Burn“, calls this the Change Function, and understanding this gives an insight into what’s likely to sell and what’s not.

Click here to read an article by Pip Coburn on the Change Function that appeared in May issue of Fast Company.

SELECT table1.* FROM table1 LEFT JOIN table2 ON table1.id=table2.id WHERE table2.id IS NULL

Now suppose you want to search not from the entire table2, but from a subset of table2 (say CityID = 1). How do you do that? You can try and see for yourself that the following SQL is not the solution:

SELECT table1.* FROM table1 LEFT JOIN table2 ON table1.id=table2.id
WHERE table2.id IS NULL
and table2.CityID = 1

Note that this is because we are looking for records that do not exist in table2 and hence cannot have a where clause on table2 data.

Read the manual page a little further, and another user comment points out:

Conditions for the “right table” go in the ON clause.
Conditions for the “left table” go in the WHERE clause,
except for the joining conditions themselves.

That makes the solution as:

SELECT table1.* FROM table1 LEFT JOIN table2 ON (table1.id=table2.id and table2.CityID = 1)
WHERE table2.id IS NULL

Happy digging!

KETL is different from Kettle, another open source ETL tool. You can read more about the similar names here at Nicholas Goodman’s blog. While Kettle is GUI oriented, KETL is scripted and probably more robust.

Read the KETL training doc to know more about its architecture and usage.

Any innovative/disruptive technolgy has first gone through a phase during which the technology is made simple and more useful for its end users. Unfortunately, BI vendors are still focussed on adding more complex features (that even their own teams don’t know how to use) instead of focusing on simplicity and productivity. And the third-party consultants sustain the con because it helps them hike their billings.

In the short run, the client suffers. But in the long run, the analytics profession suffers as a whole due to unmet promises and millions of dollars in sunk costs.

The way out? Simplicity. Read the DMReview article below for a ’simple’ way of reaping huge BI benefits.

A Simple Plan: Embracing Simplicity Can Reap Huge Business Intelligence Benefits

The true value of BI is in viewing (and nurturing) it as the cognitive base and a response model for an organization in dialog with the ‘external reality’. In that sense BI is a truly disruptive phenomenon, even though it is the logical next step for the all-round ‘digitization’ that has been taking place for the last few years.

Yet, most managers appear to be struggling to fit Analytics into their traditional repertoire instead of looking at it afresh and leveraging it like any disruptive technology should.

An interesting HBS Working Knowledge article about Disruptive Innovation that may give you some ideas: Six Keys to Building New Markets by Unleashing Disruptive Innovation : HBS Working Knowledge

Also interesting is the classification of the system environment into Rigid, Deterministic, Purposive, Heyristic, and Purpose Seeking, and the imapct of this classification on BI design.

Click here to read the article ‘A Systems Theory of Business Intelligence’.

Agile development is an approach that “cycles” through the development phases, from gathering requirements to delivering functionality into a working release. Two widely used agile development frameworks are the Microsoft Solutions Framework (MSF) and Extreme Programming (XP).

Agile development methodologies begin with ‘user stories’ which are brief 2-3 sentence problem statement. They tend to describe just enough of user requirements to get a rough idea of effort and scope. These are then used to develop multi-release plans that establish the roadmap for the team and establish the longer term vision for the team.

The actual development is done in frequent cycles of small releases (both fresh development releases as well stabilization releases). Stories are assigned to individual development cycles depending upon resources, time and requirements. This Resource-Time-Requirement triangle is actively managed through triage to make sure that release schedules are met.

Click here to read about applying Agile development methodologies to Data Warehousing projects.

The paper is targeted at financial services that are not doing analytics yet (there are quite a few of them around actually!). It sets out a road map for doing analytics by starting small to quickly start delivering business value, and then gradually taking on more complex projects.

Click here to read Delivering on the BI Promise: A Financial Services Road Map

Thanks to Dratz for letting me lift a graphic from his paper on BI as an Abstraction Architecture.