IT Architecture for the Enterprise

Semantech: IT Architecture in the Enterprise from Stephen Lahanas

Read More

Enterprise Big Data

Semantech Inc. - Mastering Enterprise Big Data - Intro from Stephen Lahanas

Read More

How to Set Trends

IT is a very trendy industry. You might say we’re more or less slaves to fashion. The priorities which are set in company boardrooms and technology departments are quite driven by the set of 'must have' capability being hyped at any given time. We’ll forgo discussion as to whether this is a good thing or not for now and focus instead on the act of setting a trend rather than merely following one.

First of all, why would someone want to be a trendsetter in IT; well here are a few potential motivations:

1. Because inventing something is fun, exciting or at the very least represents a break from the typical routine. 
2. Because identifying a trend can result in benefits to the trendsetter. I’ll emphasize “can” here because living on the cutting edge often results in some blood loss. There are risks to being new, different or the first one out of the gate.
3. Because creating your hype is and always has been good business. Not everyone can pull it off, but many of the most successful technology companies were built (at least initially) on hype.

Now, it may sound like I’m trivializing the topic a bit or perhaps implying that there is little substance behind most trends – but I’m not really. There can be a marriage behind trend-setting and true innovation if that’s the way you wish to pursue it. Obviously, some trends involve a much higher investment threshold than others. For example, inventing the next commercial quantum computing platform is going to require a lot of capital up front to pull off. However, there are lot’s opportunities hidden with the mass of existing technologies that remain unexploited as well. For most of us, the easiest path towards that exploitation is through invention of new practices.

Who defines the road ahead?

So what does that mean exactly? Practice, any practice, is the day to day approach that a professional takes to do their job. In Medicine, practice is the application of diagnostic and prognostic algorithms (rulesets and techniques) based upon understanding of a much wider body of knowledge. Individual practice in Medicine can influence the holistic practice of medicine across an entire society (or the world) through the publication of these new techniques and insights. Those new techniques if grouped together can also be viewed as trends. Some examples from Medicine of recent trends include:

• Introduction of Electronic Healthcare Records 
• Reductions in the amount and frequency of antibiotics being prescribed
• Exploitation of genetic markers to predict, diagnose and treat a wide array of illnesses

Notice that in the above examples, two of the three trends listed were in fact technology-focused. This is an important consideration when thinking about becoming a trendsetter – nearly every industry now depends on IT to provide a significant amount of its innovation. This means that IT professionals have the opportunity to set trends not only for their own industry but every other one as well. This is exciting and extremely cool if you think about it. For the first time in perhaps forever – folks who weren't trained in specialized fields are helping to redefine those fields because of the technology being applied. It’s not uncommon for an IT consultant to move across industries from one project to the next – in each one learning the business enough to determine how best emerging technology might improve it.

Maybe the best way to describe how to become a trendsetter is through an example – so I’ll draw upon one from my own experience. In 2007, I was consulting for a firm that specialized in Business Intelligence and Data Warehousing. They asked me to determine if some of their solution offerings might be cobbled together into some sort of unified practice approach. I had worked with them before and seen their solution in action and had worked in that industry space quite a lot as well. So here’s what I did:

1. I identified all of their strengths and characterized them in depth.
2. I assessed the current practice approaches in industry (circa 2007) for BI and DW and identified a list of gaps.
3. I mapped the identified strengths of the company against the industry gaps and as I suspected there were several key areas of alignment. In other words, there were things that the company was doing that wasn't commonplace in industry yet – but really needed. 
4. I took that and expanded upon it; identifying a theme for a practice approach as well as a methodology and also identified areas where some technology gaps still existed (even within the new model). 
5. This then led to a standard solutions architecture and detailed process approach based upon the theme and high level methodology. 
6. I actualized this within the company buy presenting the findings and recommendations – many of which were focused on how to blend the new practice approach within the current business model.
7. I then launched an outreach effort – through existing customers and social media. 

The practice I created for them was coined “Agile BI.” At the time, the reception was mixed. Very few in the industry were viewing Business Intelligence or Data Warehousing as something that could be considered Agile. This was before the explosion of Big Data and NoSQL solutions. The main premise I was promoting though is very close to what the industry is defining as Agile BI now:

• That BI projects can be time-boxed into clearly defined increments (you don’t have to use Scrum per se). The key is that some capability is delivered within each increment. 
• The approach was less product-focused and more capability focused. In other words, in this practice a wider variety of tools and capabilities were considered mainly because of the recognition that most environments have a wide variety of data needs / solutions.  
• That BI and the underlying data structures needed to be engineered for performance from day 1.
• That BI and DW were not IT only efforts but required very close collaboration at all stages with key business stakeholders. 
• That new modeling or architecture techniques were needed (that extended beyond 3NF ERD) to help unify management of diverse data resources. 

I recall presenting this all in Chicago in 2007 at an industry conference and receiving a lot of blank stares. This of course is where the risk comes in; it takes years and some serious marketing to go from a kernel approach to widespread industry adoption. In this particular example – that actually has occurred - ( see The TDWI Agile BI Portal ). The company I introduced the practice to did not have the resources to wholeheartedly pursue the approach and stuck to their old business model and I moved on, but not before posting several presentations online and creating a blog that may have helped contribute to the general mindshare behind the industry premise. Granted – it was an obvious and logical progression that would have likely occurred anyway - but my contributions proved to be a worthwhile exercise none-the-less.  

One of the side benefits though of helping to shape industry-wide expectations of how technology practice ought to look like is that eventually you find yourself in projects that have adopted the trend you helped to set – which in itself is fairly satisfying – especially if you really did help to promote a better way of getting things done instead of just generating more hype.

Read More

Understanding Data Warehouse Challenges

Last week we talked about some of the issues arising from the emerging field of Big Data, today we're going to explore a more mature data solution and point out some of the challenges that it's faced - and some of those challenges were instrumental in pushing the IT industry towards Big Data.

The Data Warehouse concept is built atop the notion that all data related to the enterprise can be captured and centrally or holistically managed. This is a powerful idea, yet there is more than one way to achieve that goal. The traditional view of the EDW attacked the problem from a very DBMS-centric perspective. This is primarily why EDW projects become so expensive, difficult and ultimately hard to adopt. The typical EDW approach attempted to gather all of the data related to the enterprise and place it into one massive repository structure. Whether this approach was attempted in chunks or as a “Big Bang” assault made little difference in the long the run as the byproducts of the practice were the same; those byproducts included:

• A more bureaucratic management approach to the data layer in general.
• An added degree of separation between the data owners and the data developers.
• A certain level of inflexibility in regards to how data was updated, corrected or otherwise transformed.
• An added degree of separation between database developers and data exploitation developers.
• An added degree of separation between database developers and application developers.
• An inability to quickly respond to major changes in the business.
• Dependence upon a sub-set of industry experts and equipment that is more expensive than the industry norm.
• A higher cost associated with scalability in general.

It is worth examining some of the core concepts associated with Data Warehousing a little bit closer to help understand why these outcomes tend to result from the traditional EDW approach.

DBMS Focus – At the time when EDWs became popular, other areas of data architecture were only just beginning to blossom. Today’s Business Intelligence platforms represent much more than mere reporting engines. Metadata management was only just beginning to be understood in the mid-1990’s and focus on Semantic technologies was virtually non-existent. The world according to DBMS in 1995 had a relational management system in the middle with ETL feeding into and reports coming out. This might be thought of as a three layer, stove-piped database systems view of the data architecture.

The Enterprise Single Instance – While consolidating like capabilities into marts or stores or some other ‘functional single instance’ approach has achieved quite a bit of success over the past two decades, attempting to manage all data in one structure has proven much more difficult. This is why the notion of Massively Parallel Processing (MPP) was needed to make it viable back in the 1990s. MPP in the context of proprietary hardware was expensive though and perhaps failed to recognize the power of networked processors on inexpensive hardware (i.e the Google scalability model). The other key consideration here was the added steps that were needed in order to make such a system perform within reasonable parameters. So, the single instance enterprise faced and still faces major hurdles in terms of costs, manageability and performance. 

Data is no longer confined within the context of single systems

EDW Fallacies
If we were to directly challenge the core EDW assumptions and illustrate the fallacies associated with the philosophy, our list would resemble the following:

• The Business will remain static over a relatively long period of time.
• The Enterprise will remain static over a relatively long period of time.
• That source data and data exploitation should not be managed synergistically, in other words that Decision Support or Business Intelligence solutions built on top of EDW source data should be viewed as separate, albeit related efforts.
• That the data layer and the application layer can or should be viewed or designed separately.
• That computer Hardware would not catch up to the processing load – i.e. that the data layer would always require specialized Massively Parallel Processing (MPP) in order to manage very large quantities of data. Furthermore, this assumption also implied the data would remain in a single instance data source. So instead of parallel processors deployed in specialized equipment, Big Data now uses the cheapest processes / equipment possible in a commodity approach with data spread out in sets of distributed file systems. In fact this has been take even further as this week the US Government announced the completion of the world's most powerful supercomputer. It achieved all of its latest gains by using commodity hardware (in this case, GPUs, game video processors widely available on the market).
• That network architecture, data architecture, application / SOA architecture and enterprise architecture are separate.
• That the Internet (Cloud) would not represent a viable mechanism for connecting to distributed data sources.
• That unstructured data was not as valid as structured data (mainly because no mechanism existed to incorporate into the traditional database management approaches).
• That most major transformations need to occur before data is placed into the primary storage / management entity (i.e. DBMS, warehouse).
• That there is a single version of the truth, period. This is perhaps the biggest fallacy behind all data warehouse, MDM and Governance solutions. Data can be managed, but it is dynamic and all always will be. Viewing data as incontrovertible, orthodox truth immediately eliminates much of the value that data otherwise provides. Situations change, and every stakeholder views the whole from their unique perspectives. Yet, there can still be order in a relativistic environment (much as there is in the real world).  This doesn't mean data cannot be standardized or managed - it merely takes into consideration the inevitable evolution that will occur.

Copyright 2012  - Technovation Talks, Semantech Inc.

Read More