Trends in Data warehouse and Data Visualization tools


Data warehouse for any enterprise is always in an evolving state. We seem to be now maturing from report only to analytical phase and therefore one step closer to our dream of real time data warehouse. Every change in phase brings along with it new sets of challenges and therefore, often, new sets of tools and technologies.
As we move from reporting to analytical phase, BI giants such as Cognos, MSTR, BO, OBIEE, Hyperion seem to be getting more and more obsolete .This is where we welcome data visualization tools like QlikView, Corda, Tableau, Panoptican, Spot Fire and Tibco.
Advantages of these tools are very much in your face.



  •  They have very less build up time i.e. to say you can complete your first report within two days of installation (Including the learning phase). This is preferable as against traditional reporting tools, where you would spend considerable time building metadata layer.(MSTR claims you can bring down a 30-day activity to 1 hour).
  •  They are visually very appealing with very good integration for Web Services .Demos that I have seen for Tableau and QlikView are very engaging.
  •  They have on-the-fly filtering and roll up/drill down facility.Once a user gets to see the final report, he can select values across different dimensions ( like time ,location ..) and see data getting updated in the webpage almost instantaneously ( if not instantaneously). This makes the concept of reports with traditional filters and prompts very archaic. Therefore, a very nicely written generalized query can remove need to reuse the same set of data in different reports with varying filters/prompts. This reduces your need for metadata layer to an extent.
  • Since they provide reasonable flexibility for ad hoc processing by a business user (Self served BI reports); these can reduce your need for ad hoc team by a considerable amount.(Not a good sign for Consultants  ).
  • Although I do not know the actual cost, I am inclined to believe that these tools would be much more economical compared to traditional BI.
  • It inherently seems to be supportive of the latest fad ; Mobile BI .Being able to see reports on Tablets/Smart phones /Kindles (OK, I don’t know about Kindles ,but others still apply  ),

One would believe that providing this level of flexibility with data would impact performance; however, they seem to be creating another layer between the database and report output; where this data is cached and managed rather ingeniously. So, when a user drills down or applies filter, the data is fetched form cached memory instead of the database .Of course, this means you have a possibility of reading stale data. However, most of the data warehouses are not expected to be real time, so this is not usually an issue.
Is this an indication of paradigm shift in BI?
Yes, very much. In fact, this seems to be precisely the reason MSTR has come up with Micro Strategy Visual Insight; its own DV tool.


More on it in the following links:


http://www.pcworld.com/businesscenter/article/225619/microstrategy_takes_aim_at_selfservice_bi.html
http://www.microstrategy.com/visual-insight/


However considering strong integration servies BI tools have with ETL. For instance BO and SAP or Entire hyperion suite and obiee.BI tools are not to disappear any time soon.But DV tools will have a definite impact on their growth.




Note to Self : Update blog post with relevant snaps to demonstrate features.

Comments

Post a Comment

Popular posts from this blog

Macro to generate Mload Script

DROP MACRO MLOAD_EXPORT REPLACE MACRO MLOAD_EXPORT ( DB VARCHAR(100), TB VARCHAR(100), PATH VARCHAR(100), FS VARCHAR(100) ) AS ( SEL 1,CAST(1 AS BIGINT), CAST('.DATEFORM ANSIDATE; .LOGON server/username,password; .LOGTABLE USR_WORK.ML_UMS_' || :TB || '_1; DROP TABLE USR_WORK.WT_UMS_' || :TB || '_1; DROP TABLE USR_WORK.ET_UMS_' || :TB || '_1; DROP TABLE USR_WORK.UV_UMS_' || :TB || '_1; .ROUTE MESSAGES WITH ECHO TO FILE ''' || :PATH || '.ldrlog'' ; .BEGIN IMPORT MLOAD TABLES ' || :DB || '.' || :TB || ' WORKTABLES USR_WORK.WT_UMS_' || :TB || '_1 ERRORTABLES USR_WORK.ET_UMS_' || :TB || '_1 USR_WORK.UV_UMS_' || :TB || '_1 ERRLIMIT          0 CHECKPOINT     100000 TENACITY 10000 SESSIONS        10 SLEEP 6 ; .LAYOUT InputFileLayout; ' AS VARCHAR(6000)) FROM SYS_CALENDAR.CALENDAR WHERE CALENDAR_DATE=DATE UNION SEL 2,CAST(COLUMNID AS BIGINT), CAST('.FI
Read more

Finding Skewed Tables in Teradata

       Skewed Tables in Teradata                                                                  Teradata distributes its data for a given table based on its Primary Index and Primary Index alone .If this Primary Index is not selected appropriately it can cause performance bottle necks. Following are general parameters we should consider while creating Primary Indexes       1:Access Path       2:Volatility       3: Data Distribution        Volatility is usually not an issue in a well designed database .That is to say we do not expect update clauses updating the primary index itself. This usually leaves us with data distribution and Access Path.       Access Path implies particular column ( set of columns ) are always used in join conditions .Advantage of using these columns as PI is that it will avoid redistribution of data during join .( One of the most expensive operation for teradata ) and therefore can reduce usage of Spool files.        Data distribution implies
Read more

The Secret behind A B I N I T I O.

Background on Abinitio : Abinitio is considered one of the major ETL players  in line with informatica ,datastage ,SSIS ( if it can be considered major ETL player ;-) ) .In fact, in most respects it is considered better than all of the above . It has user interface and debugging simpler than Informatica  ( one of the positive points about informatica ) ,It has parallelism which data stage may still take years to reach . Superiority : (HearSay ) It has the ability to convert a 6 people ,6 month ETL project to 2 month 4 people ETL project . (Sounds hyperbole ,But yet it has been established in many cases that it takes half the development time compared to INFA with much better quality of code ) . Other statistics suggest that if you call Abinitio ,there is 75% chance you will be talking to a PhD holder.In other words it is considered to be ingenously simple. Secrecy : In spite of all this there is very little you would know about it (If at all ) . Reason : It is so because Abi
Read more