Understanding Data Architecture

Someone asked me what at first sounded like a very straightforward question earlier this week; "what is Data Architecture" - or more precisely, what does it mean to you. Usually, I'm not usually at a loss for words when it comes to expounding upon IT Architecture related topics - but it occurred to me at that moment that my previous understanding of what Data Architecture really represents is or has been a little flawed or perhaps just outdated. So I gave a somewhat convoluted and circumspect answer.

Where does Architecture fit within this picture?

The nature of what's occurring in the Data domain within IT is itself changing - very quickly and somewhat radically. The rise of Big Data and proliferation of User Driven discovery tools represents quite a departure from the previous more deterministic view of how data ought to be organized, processed and harvested. So how does all of this effect Data Architecture as a practice within IT (or more specifically within IT Architecture)?

But before we dive into the implications of the current revolution and its subsequent democratizing of data, we need to step back and look again the more traditional definitions as to what Data Architecture represents. I'll start with a high level summary view:

Traditional Data Architecture can be divided into two main focus areas; 1 - the structure of the data itself and 2 - the systems view of whatever components are utilized to exploit the data contained within the systems. Data in itself is the semantic representation or shorthand of the processes, functions or activities that an organization is involved with. Data has traditionally been subdivided (at least for the past several decades) into two categories; transactional and knowledge-based or analytic (OLTP vs. OLAP). 

Now we'll move to a traditional summary definition of Data Architecture practice:

Data Architecture is the practice of managing both the design of data as well as of the systems which house or exploit that data. As such, this practice area revolves around management of data models and architecture models. Unfortunately, the application of Governance within this practice is sporadic and when it does occur is often split into two views: governance of the data (models) and governance of systems (patterns and configurations). 

So, that seems to be fairly comprehensive; but is it? Where does Business Intelligence fit in - is it part of the data management or system management - is it purely knowledge focused or does it also include transactional data? For that matter, do Data Warehouses only concern themselves with analytic data or can they be used to pass through transactional data to other consumers? And isn't Big Data both transactional and analytic in nature? And BTW- how do you model Big Data solutions either from a systems or data modeling standpoint? Now - we start to begin seeing how things can get confusing.

We also need to take into consideration that there has been an attempt made to standardize some of this from an industry perspective - it's referred to as the Data Management Book of Practice or DMBOK. I think in some ways it's been successful in attempting to lay out an industry taxonomy (much like ITIL did) but not as successful in linking that back into the practice of Data Architecture. The following diagram represents an attempt to map the two together...

There isn't a one to mapping between DMBOK and data architecture practice, but it's close

One of the areas that the DMBOK has fallen short is Big Data; my guess is that they will need to rethink their framework once again relatively soon to accommodate what's happening in the real world. In the diagram above, we have a somewhat idealized view in that we've targeted a unified governance approach for both data modeling and data systems.

Let's take a moment and discuss the challenges presented by the advent of new Big Data and BI technology. We'll start with BI - let's say your organization is using Oracle's BI suite - Oracle Business Intelligence Enterprise Edition (OBIEE). Within OBIEE you have a more or less semantic / metadata management tool called Common Enterprise Information Model (CEIM). It produces a file (or files) that maps out the business functionality of all the reports or dashboards associated with the solution. Where does that fit from an architecture standpoint? It has a modeling like interface but it isn't a 3rd normal form model or even a dimensional model. It represents a proprietary Oracle approach (both as an interface and modeling approach). It allows you to track dimensions, data hierarchies and data structures - so it is a viable architecture management tool for BI (at least for OBIEE instantiations). But some traditional Data Architecture groups would not view this as something the architects would manage - it might handed off to OBIEE administrators. This situation is not unique to Oracle of course, it applies to IBM / Cognos and other BI tools as well and there's a whole new class of tools that are completely driven by end users (rather than structured in advance from an IT group).

Now let's look at Big Data. Many of the Big Data tools require command line interface management and programming in order to create or change core data structures. There is no standard modeling approach for Big Data as it encompasses at least 5 different major approaches (as different say as 3NF is from Dimensional). How does an architecture group manage this? Right now, in most cases it's not managed as data architecture but more as data systems architecture. The problem here is obvious; just as organizations have finally gained some insight into the data they own or manage - a giant new elephant as entered the room. How is that new capability going to impact the rest of the enterprise - how can it be managed effectively?

Back to the original question - what is Data Architecture. I'd like to suggest that the practice of Data Architecture is more than the sum of its traditional activities. Data Architecture is the practice of understanding, managing and properly exploiting data in the context of problems any given organization has to solve. It is not limited by prior classifications or practice but has a consistent mandate to be able to represent and hopefully govern in some fashion data as an asset (internally or shared collaboratively). Data Architecture as we know is going to change quite a bit in the next two years and that's a very good thing.



Copyright 2013, Stephen Lahanas



#Semantech
#StephenLahanas

Read More

The Value of Architecture Assessments

While many people are becoming somewhat familiar with IT or Enterprise Architecture, relatively few know much about Architecture Assessments. This is unfortunate given the significant value proposition such exercises provide. For example, had one or more Architecture Assessments been provided for Healthcare.gov during the course of the project; it is unlikely that the Obama administration would have been surprised by the current mess it's now facing.

Problem Space analysis is one of the techniques used with assessments - it can apply
both to the business and technical aspects of a project

An Architecture Assessment is also different than other traditional Architecture activities in that the expectation is generally that third-party personnel are more likely to perform them. The reasons for this include the following:

1. Assessments are one of the key tools involved in IT oversight activities (sometimes referred to as Independent Validation and Verification or IV&V).
2. It is more likely that an accurate assessment can be obtained by architects / investigators without a vested interest in the project. 
3. The skillset of the person doing the assessment is critical - it needs to be an architect and not merely a technical or product expert. This is the only way to ensure that all options / alternatives are properly considered / assessed. 

So what exactly is an Architecture Assessment? A typical assessment tends to include the following categories of activity:

• Information Gathering  
• Design & Project Review
• Design & Project Recommendations

An assessment also typically includes one or more types of specific analysis as well:

• Analysis of Alternatives
• Root Cause Analysis
• Problem Space Mapping & Resolution 

One way to capture alternatives is through Decision Trees.

Perhaps the most important function that an Architecture Assessment can provide is a mechanism to challenge assumptions and combat complacency. One of the main reasons that IT projects fail today is because there is generally no incentive or expectation to raise issues or problems. Rather than being viewed as a healthy activity - identification of the problems is feared in itself and thus ensures even more pain later on when the issue finally surfaces (which they always do).

When compared to the cost of the rest of a typical IT project (and any potential loss or cost overruns associated with problems that aren't identified or managed in a timely fashion), a relatively brief assessment exercise is generally less than 1% of total cost yet could make the difference between success or failure...



Copyright 2013, Stephen Lahanas


#Semantech
#ITarchitecture

Read More

What Does an IT Architect Do?

This is an interesting question for anyone thinking about working as an Architect but also for anyone else working in IT because there isn't a consistent set of definitions regarding architects within the industry. Architects who do more or less the same things with the same skills could variously be referred to as:

• Enterprise Architects
• Solution Architects
• Project Architects
• IT Architects
• Technical Architects
• Data Architects
• Application Architects
• Business Architects
• Cloud or SOA Architects
• Security Architects

There are many types of architects, yet they all share certain characteristics...

And the list goes on. What if anything differentiates these roles (1)? More importantly, what elements do these roles have in common (2), if any? Not too long ago in the history of IT, few people used the term Architect to describe anyone. Why has this changed (3) – we've gone from no architects to more than a dozen flavors in perhaps 15 or so years?

Let’s answer some of these questions:

1. The Architecture roles (listed above) are generally differentiated by some level of solution specific expertise in one area or another. This could involve methodology, toolsets, skillsets or even industry domain knowledge.
2. An Architect is not or at least should not be tied to one specific element of expertise – if he or she is focused in only one area then they become a Subject Matter Expert (SME) and not an Architect. This is an important and practical distinction because technology keeps changing – today’s stack will not be the same as tomorrows’. Architects must understand the entire stack as it evolves and change skill sets as that evolution occurs. All Architects are designers. All Architects are problem solvers. All Architects are by nature also systems and business analysts. 
3. The genesis of the Architecture role in IT is directly related to the rapid decentralization and added complexities associated with the PC and Client Server revolution (and everything else that has occurred since then). In other words, as IT environments and systems became more complex, it was apparent that a “complexity manager” was required. That person is the Architect. 

Why would we necessarily refer to a complexity manager as an Architect? The metaphor as designer is obvious – but just as important is the idea that the Architect is the one who has the vision and understanding to see how all of the various pieces ought to fit together. An Architect, in either realm is by nature also an integrator.

So, how does an IT Architect differ from a traditional Architect (the folks who design blueprints for buildings)? Perhaps the most interesting difference between the two roles is that there is often an expectation within IT that the Architect or Designer is also the Builder. In the world of brick and mortar construction, it is very rare to see builders follow a career path from hanging drywall and pouring foundations to drafting the design for the entire building. Yet, in Information Technology it is relatively common to see developers become architects.

There is a good reason why this happens in IT but there is also a problematic result as well. The reasons why it happens are because:

1. the architecture career path is still unclear and we have to get Architects from somewhere and 
2. unlike with buildings, Architects in IT need to understand more of the technology associated with what gets built. 

We need that understanding because IT is much more dynamic than building design – we ‘reinvent’ our industry about every 5 to 10 years now. Some people still consider Frank Lloyd Wright’s work to be cutting edge and he’s been dead for more than 50 years – clearly IT and traditional architecture move at a different pace.

The problematic outcome of this situation is that many Architects tend to view the design and analytic processes associated with architecture to be inferior to ‘just building something’. This viewpoint more or less contradicts the true mission of what Architecture is supposed to do and the problem that it solves. In other words, Architecture has arisen to manage complexity; yet rapid build with minimal analysis is the number one culprit behind increasing complexity in the first place. This type of Architect could be referred to as an Architect in name only because it generally implies that they would not be practicing many of the key attributes associated with Architecture. This also potentially opens up a larger debate regarding Agile versus non-Agile IT, and that debate will never go away. The important takeaway from that debate though is when you reach the systems of system level; architecture becomes increasingly important.

Contrary to popular belief, Architecture and Agile Methodology actually complement one another.

There is a another consideration in understanding the role of the IT architect as well – without industry standard expectations in relation to what the Architect role actually represents – there can be wild inconsistencies across or even within organizations that utilize architects.  So, how could we solve this dilemma? Here are some suggestions:

1. Develop an industry standard set of role descriptions for IT Architecture. (there are groups that have developed standards for Enterprise Architects, but that is entirely too narrow to handle the larger set of expectations associated with IT architecture).
2. Ensure that any Architect in any role, anywhere – is given the top level training or expectations that are common across all architecture first (before drilling down). 
3. Help foster the distinctions between lead developers, tech leads, SMEs and Architects. This will help organizations determine when they really need an Architect as opposed to one of the other roles. If the roles are mixed it is highly likely that one part or the other of the combined role is going to get shortchanged – and that could lead to a number of unforeseen consequences. 

There are several other key attributes that help to distinguish IT Architects from other roles in IT; these include:

• Architects are often asked to act as liaison between other solution stakeholders. Sometimes Architects even become the official solution advocates.
• While sometimes asked to be advocates, Architects also tend to be the key resource in determining when a solution needs to be dropped. An Architect must be impartial when making such decisions.  
• Architects are complexity experts or managers – in other words typically Architects are dealing with “Systems of Systems” scenarios. Thus, the Architect has to be concerned not just with how the solution will operate in its own context, but how it will function in the context of the larger ecosystem. 
• Architects act as honest brokers in being able to question assumptions and drive change in order to mitigate potential risks. While other roles may be involved in this; typically Architects have the best vantage point to deal with it.
• Architects are change agents – more-so than any other IT role. Architects are asked to either envision or evaluate new technology and lead the move towards it.

The IT Architect is a relatively new role; it is often interpreted differently but it is uniquely positioned to become ever more important. As this role continues to help evolve the IT landscape, it too will evolve.


Copyright 2013,  Stephen Lahanas

#Semantech

Read More