How to Create an Enterprise Data Strategy

I work as an IT Architect. One of the more interesting things I get asked to do on occasion is to create Strategies in particular technology areas. This represents the "high-level" side of the typical IT Architecture duties one tends to run into (if you're interested in more IT Architecture related topics check my new blog here - The IT Architecture Journal). A very popular strategic focus across industry lately is Data Strategy. In this post, I will try to explain why it has gotten so popular and some of the fundamental aspects of actually producing one.

Data Strategy, Defined
Data Strategy is the collection of principles, decisions, expectations as well as specific goals and objectives in regards to how enterprise data and related data systems and services will be managed or enhanced over a specified future time period. As an actual artifact, Data Strategy is usually manifested as a document, but is not limited to that format.

A Data Strategy is usually conducted in conjunction with some IT Portfolio planning process. In optimal situations, portfolio decisions and follow-on data-related projects can be mapped directly back to goals and objectives in the Data Strategy.

Why Data Strategy is so popular
Organizations across the world have become more aware of the need for greater attention to data related issues in recent years. Some of this has been driven by collaborative industry initiatives through groups like DAMA (the Data Management Association) and the resulting Data Management Book of Practice (DMBOK). Other drivers include the near-flood of new data technologies released over the past decade as well as the exponentially growing quantity of data out there.

So, what does having a Strategy give actually give you?

What it often provides, if used properly, is both a set of shared expectations as well as a clear path for actualization of those expectations. The Data Strategy allows organizations to deliberately decide how best to exploit their data and to commit to the major investments which might be necessary to support that. This, when contrasted with an hoc and decentralized technology evolution scenario presents a much easier picture to grasp. And it also at least implies a situation that will be easier to predict or otherwise manage. It is that promise of manageability that makes creating a Data Strategy so attractive.

Elements of a Typical Data Strategy
The following mindmap illustrates some of the common elements that you'll find in many data strategies. One noteworthy item in this diagram is the idea of sub-strategies (which can be split off into separate documents / artifacts) ...

The Top 7 Considerations for Data Strategy
While there are many more things to keep in mind, I've tried to distill some of the most important considerations for this post...

1. The strategy should take into account all data associated with the enterprise. This may sound obvious but in fact it isn't really that obvious. Many organizations explicitly separate management of dedicated data systems from other systems which may have data in them but aren't strictly just DBMSs or reports, etc. For example, there may be state data in small data stores associated with a web-based application that supports an online form / application - the data structures supporting the completion of the form may be different than the ones which collect the completed form data. However, all data, in all applications regardless of where it may be located or how or it is used must be considered.  
2. There generally needs to be an attempt to define an organizational 'lingua franca' - or a common semantic understanding of data. There are many ways this might be achieved, but it is important that this included within the strategic plan.
3. The Strategy cannot be entirely generic, even if one of the most vital objectives is some type of industry-driven standardization. Wholly generic plans are usually less than helpful.
4. The Data Strategy must be presented within a larger context. What this means is that there needs to be an expectation that the Strategy will indeed be the precursor to other activities which ought to be able to map back to it for traceability purposes. 
5. The Data Strategy needs to have sufficient detail to be meaningful. If it is too high-level it becomes merely an elaborate Mission Statement. The expectation behind any Strategy or plan is that it be actionable. 
6. The Data Strategy ought to be need or capability based - not product or Hype focused. 
7. There ought to be a way to measure success 'built into' the Data Strategy. This can come in the form of basic service level expectations or business outcomes or both. 

What goes into one Data Strategy versus another can be radically different from group to group. If you have a Social Media company your needs will be quite different than the US Coast Guard for example - but both will likely need their own Data Strategy.

Copyright 2014, Stephen Lahanas

#Semantech
#StephenLahanas
#TechnovationTalks

Read More

Understanding Data Architecture

Someone asked me what at first sounded like a very straightforward question earlier this week; "what is Data Architecture" - or more precisely, what does it mean to you. Usually, I'm not usually at a loss for words when it comes to expounding upon IT Architecture related topics - but it occurred to me at that moment that my previous understanding of what Data Architecture really represents is or has been a little flawed or perhaps just outdated. So I gave a somewhat convoluted and circumspect answer.

Where does Architecture fit within this picture?

The nature of what's occurring in the Data domain within IT is itself changing - very quickly and somewhat radically. The rise of Big Data and proliferation of User Driven discovery tools represents quite a departure from the previous more deterministic view of how data ought to be organized, processed and harvested. So how does all of this effect Data Architecture as a practice within IT (or more specifically within IT Architecture)?

But before we dive into the implications of the current revolution and its subsequent democratizing of data, we need to step back and look again the more traditional definitions as to what Data Architecture represents. I'll start with a high level summary view:

Traditional Data Architecture can be divided into two main focus areas; 1 - the structure of the data itself and 2 - the systems view of whatever components are utilized to exploit the data contained within the systems. Data in itself is the semantic representation or shorthand of the processes, functions or activities that an organization is involved with. Data has traditionally been subdivided (at least for the past several decades) into two categories; transactional and knowledge-based or analytic (OLTP vs. OLAP). 

Now we'll move to a traditional summary definition of Data Architecture practice:

Data Architecture is the practice of managing both the design of data as well as of the systems which house or exploit that data. As such, this practice area revolves around management of data models and architecture models. Unfortunately, the application of Governance within this practice is sporadic and when it does occur is often split into two views: governance of the data (models) and governance of systems (patterns and configurations). 

So, that seems to be fairly comprehensive; but is it? Where does Business Intelligence fit in - is it part of the data management or system management - is it purely knowledge focused or does it also include transactional data? For that matter, do Data Warehouses only concern themselves with analytic data or can they be used to pass through transactional data to other consumers? And isn't Big Data both transactional and analytic in nature? And BTW- how do you model Big Data solutions either from a systems or data modeling standpoint? Now - we start to begin seeing how things can get confusing.

We also need to take into consideration that there has been an attempt made to standardize some of this from an industry perspective - it's referred to as the Data Management Book of Practice or DMBOK. I think in some ways it's been successful in attempting to lay out an industry taxonomy (much like ITIL did) but not as successful in linking that back into the practice of Data Architecture. The following diagram represents an attempt to map the two together...

There isn't a one to mapping between DMBOK and data architecture practice, but it's close

One of the areas that the DMBOK has fallen short is Big Data; my guess is that they will need to rethink their framework once again relatively soon to accommodate what's happening in the real world. In the diagram above, we have a somewhat idealized view in that we've targeted a unified governance approach for both data modeling and data systems.

Let's take a moment and discuss the challenges presented by the advent of new Big Data and BI technology. We'll start with BI - let's say your organization is using Oracle's BI suite - Oracle Business Intelligence Enterprise Edition (OBIEE). Within OBIEE you have a more or less semantic / metadata management tool called Common Enterprise Information Model (CEIM). It produces a file (or files) that maps out the business functionality of all the reports or dashboards associated with the solution. Where does that fit from an architecture standpoint? It has a modeling like interface but it isn't a 3rd normal form model or even a dimensional model. It represents a proprietary Oracle approach (both as an interface and modeling approach). It allows you to track dimensions, data hierarchies and data structures - so it is a viable architecture management tool for BI (at least for OBIEE instantiations). But some traditional Data Architecture groups would not view this as something the architects would manage - it might handed off to OBIEE administrators. This situation is not unique to Oracle of course, it applies to IBM / Cognos and other BI tools as well and there's a whole new class of tools that are completely driven by end users (rather than structured in advance from an IT group).

Now let's look at Big Data. Many of the Big Data tools require command line interface management and programming in order to create or change core data structures. There is no standard modeling approach for Big Data as it encompasses at least 5 different major approaches (as different say as 3NF is from Dimensional). How does an architecture group manage this? Right now, in most cases it's not managed as data architecture but more as data systems architecture. The problem here is obvious; just as organizations have finally gained some insight into the data they own or manage - a giant new elephant as entered the room. How is that new capability going to impact the rest of the enterprise - how can it be managed effectively?

Back to the original question - what is Data Architecture. I'd like to suggest that the practice of Data Architecture is more than the sum of its traditional activities. Data Architecture is the practice of understanding, managing and properly exploiting data in the context of problems any given organization has to solve. It is not limited by prior classifications or practice but has a consistent mandate to be able to represent and hopefully govern in some fashion data as an asset (internally or shared collaboratively). Data Architecture as we know is going to change quite a bit in the next two years and that's a very good thing.



Copyright 2013, Stephen Lahanas



#Semantech
#StephenLahanas

Read More

Integrating Design & Architecture

We're going to take a short break from Physics and dive back into IT - actually pretty soon we will see them intersect in our discussions - but for now we're going to expand on a topic I introduced on Dice.com last year: Integrated Design.

Many people view design work and architecture within IT as separate disciplines and perhaps as many different disciplines. It's my firm belief that all design-related effort (and architecture most definitely falls into that bucket) are related and contextual. The challenge of course is finding a way to integrate these activities - first from a process perspective and eventually through some sort of automation. One of the best ways to achieve this is through the development of an Integrated Solutions Framework.

One of the interesting characteristics of most Enterprise Architecture frameworks is that they are built atop meta-models. Those meta-models are in fact usually a 3NF (3rd normal form) relational data model (or ERD – entity relationship diagram). Having the models allows for the ability to deploy design artifacts to architecture repositories – it also provides a metadata framework that allows for reporting based upon the information resident within the designs. But most importantly perhaps, this illustrates how important it is to have a logical “mapping framework” for all information related to design. While using one of those frameworks within an actual software solution dedicated to architecture management is nice, it isn't necessary. One can recreate the logical framework as an organizational construct and still receive quite a number of benefits from it.

Here is a visual representation of what that standard framework might look like...

A prototypical framework for integrating design and architecture

In the following sections, I will provide potential benefits and explain the framework in more detail.

Benefits:

• This framework provides a clear linkage between specific elements of the architecture/design and requirements.
• This framework allows for consistent and logical descriptions of all solution elements across and within a given organization (and is scalable across organizations as well).
• This framework allows for management of all non-architecture designs alongside formal architecture deliverables. This is extremely important as in most organizations the majority of IT design is not captured as architecture (and in some organizations, none is).
• This framework can be used with no automation, some automation or total automation – it is also easily translatable to most EA frameworks.
• This framework can be integrated with / into any solution methodology and any standards management paradigm.

Levels (vertical):

1. Enterprise
2. Segment
3. Capability
4. Detail

Concepts (horizontal):

1. Design Activity or Description
2. Design Deliverable
3. Example/s

Definitions:

• Core / Holistic Architecture – The Holistic Architecture can exist potentially at several levels – it is roughly equivalent to an Enterprise Architecture (EA). Like an EA, the Holistic Architecture can encompass one or many domains and organizations. The main idea is that within some defined boundary the Holistic Architecture covers the comprehensive solution in question.

• Architecture Tier - An Architecture Tier tends to represent a logical layer within a holistic solution – for example this could be the shared Cloud Hosting environment that all other elements of the solution reside within. 

• Reference Architecture– A Reference Architecture can be a tier (more horizontal) or it could be a vertical segment (crossing tiers) within the larger Holistic solution. For example, a BPM workflow capability may stretch from to the infrastructure level to the User Interface and be considered the standard approach towards providing workflow capability to that organization (thus it is a vertical segment).  

• Core Capability – A capability is often associated with a slightly lower level of detail than the Reference Architecture. An example of this might be a BPMN design tool for business users to create their own workflow. Within the example Reference Architecture listed above, the design tool is located on one of several servers and provides its own unique interface that may not be generally available to the rest of the users in the presentation layer (for the BPM / workflow tool). Capability definition is used to facilitate requirements development and Use Case modeling. A Capability in this framework approach is roughly equivalent to a complex pattern.

• Complex Pattern – A Complex Pattern simply means that more than one pattern is being employed. Although allocation of scope to patterns is a somewhat subjective exercise, the goal within the paradigm presented here is provide as narrow of a pattern as possible in order to help drives specific decisions regarding solution allocation and configuration standards. The above Capability example represents a complex pattern as there are several distinct elements to the workflow mapping solution (each of which may involve different options to realize). 

• Simple Pattern – A Simple Pattern is the lowest level of architecture or design control; in other words this is where we specify the details for how the solution ought to work. For example, using the ongoing example, we might assign separate Simple Patterns for the Design UI approach for the modeling tool based upon whether it is being delivered to a PC or a mobile device. 

• An important consideration here is the notion that an organization can have more than one methodology; and in fact many organizations now have at least two, a traditional Waterfall approach and an Agile approach of some type. More complex organizations may have any number of methodologies to deal with; for management of Data Centers ITIL could be considered a methodology, for government Acquisition DoD 5000 is used, for data-centric projects many groups create approaches based upon DAMA’s DMBOK (Data Management Body of Knowledge) and so on. The design mapping described and illustrated below could be applied to any / all of those situations. 

copyright 2013, Stephen Lahanas

Read More

The Trouble with "Big Data"

How can there be trouble with one of the two biggest trends in IT you ask? Well, perhaps from a hype and marketing perspective there isn't any trouble yet. But from an expectations perspective, the trouble began nearly two years ago and has only gotten worse. And it starts in a name, it sounds simple, but is it?

Can you define "Big Data" and if so would your define match an industry standard expectation?

What is Big Data, anyway? Well, this is where the trouble begins - it means something different to a fairly diverse set of interests. For some, Big Data implies use of a parallel processing paradigm (which BTW has been used for more than a decade in Data Warehousing as well), use of commodity hardware and a clever algorithm created by Google about a decade ago to help index the web. Much of this is now combined with the use of "Hadoop," although use of Hadoop doesn't always imply that companies will follow the same hardware path as some of the giants who pioneered the paradigm. In fact, more of than not the real market for commodity hardware is moving to the Cloud. But wait, aren't we talking about Big Data? What's the relationship between Big Data and the other biggest trend in IT today, Cloud Computing? Are they really two separate trends or variations of the same trend? The answer to that question is - who knows.

There are some other problems with Big Data; let's review them:

1. It seems to encompass a wide range of emerging technologies, such as storage, parallel processing, cloud technology, high performance discovery, new DBMS paradigms and more. 
2. The Use Cases for Big Data tend to blur into the same set of Use Cases for most enterprise data related functions. This wasn't always the case - the original Google exploitation its technology was fairly narrow and unique to its business model / mission. It sill isn't entirely clear how smaller enterprises will harness the newer Big Data capabilities - that clarity is vital - especially in regards how to integrate within the existing ecosystem.
3. There is no universally accepted definition for what it represents, but just as important, there is no recommended solution approach or set of approaches or even a recommended solution methodology. The largest IT trade group dedicated to Data Management, DAMA, has barely scratched the surface as to how integrate Big Data within the larger set of Data Management activities. Or should we assume that Big Data will somehow eventually swallow all of rest of what we were viewing as Data Management?

Let's step back in time for moment. Back in early 1999, I attended a technology conference in Washington D.C. that was convened to assess emerging technology trends for the next decade and beyond. One of the most interesting discussions that occurred during their main panel revolved around a question on how much bandwidth or data would be utilized in coming years. Recall, that in 1999, having a Terabyte of memory in a DBMS was  big deal and few if anyone had DSL like speeds for Internet access. The majority of the panel did not see any explosive growth happening in the foreseeable future. I disagreed - I countered that the demand had already been pent up and that a torrent of Digital content and communication would explode as soon as the hardware prices and bandwidth allowed. It's this exponential growth in data that the proponents of Big Data expound upon a lot these days (supposedly 2/3 thirds of all data ever created was generated in the last two years).

Well, guess what - that exponential data growth was merely a drop in the bucket to what's coming. And if that is truly the case, then we have to ask ourselves what this really means. Sure, we needed more affordable hardware and more affordable software to handle volume; we needed better algorithms and architecture to handle performance. The thing is though, we still haven't defined what this all means in relation to how we manage the enterprise. We've still got and we continue to support all sorts of legacy architectures and approaches - and now we've been handed a whole new set of challenges. But those challenges aren't just focused on bigger, cheaper, faster - we also have to deal with smarter, integrated and targeted. And we also have to become a bit more visionary when it comes to imaging what we can and should do with emerging worlds of data and that may take us right back to another set of technologies that have been emerging over the past decade right alongside Big Data - Semantic Technology. 
 
So, let's ask ourselves again:

1. Is Big Data about handling larger volumes of data faster?
2. Is Big Data about making Data Management more efficient / less expensive?
3. Is Big Data about harnessing the Cloud and Storage?
4. Is Big Data about expanding Data Discovery to cover ever-increasing sets of data?
5. Is it all of the above and / or something else?

Perhaps it's time for so more or better definition. Without better definition it will likely be to understand what the ROI is that you're shooting for or whether your organization is actually achieving it.

Copyright 2012, Semantech Inc. All Rights Reserved

Read More