IT Architecture for the Enterprise

Semantech: IT Architecture in the Enterprise from Stephen Lahanas

Read More

Revisiting Agile Business Intelligence

The other day the TDWI (the Data Warehouse Institute) sent me a brochure highlighting Agile BI workshops and seminars. Here's how they define it:

"Agile business intelligence addresses a broad need to enable flexibility by accelerating the time it takes to deliver value with BI projects. It can include technology deployment options such as self-service BI, cloud-based BI, and data discovery dashboards that allow users to begin working with data more rapidly and adjust to changing needs.

To transform traditional BI project development to fit dynamic user requirements, many organizations implement formal methodologies that utilize agile software development techniques and tools to accelerate development, testing, and deployment. Ongoing scoping, rapid iterations that deliver working components, evolving requirements, scrum sessions, frequent and thorough testing, and business/development communication are important facets of a formal agile approach. "

Now I found this very interesting given it's something I have been advocating for some time. Although, the definition above left me a bit concerned that what in fact is being suggested is merely the adoption of Agile methodology with minimal regard to Business Intelligence architecture (we've been given a laundry list of related solutions with not clear idea of how they integrate). More importantly, the heavy focus on the development methodology leaves out what we considered the most important aspect of Agile BI (when we first presented this back in 2007) - the end user and how they are integrated into the development process and /or how they drive the very structure of BI by defining it "on the fly" themselves (this goes beyond data discovery).

We presented this in the Fall of 2007 in Chicago

Agile BI must encompass a wider architectural approach...

Since 2007, a number of tools have come out that specifically answer this end-user consideration. A good example of this is Tableau (which markets itself as "visual analytics for everyone"). So on the one hand it is both gratifying and exciting to see that Agile concepts are being extended to Data Architecture and that new products are being introduced to help bridge the gap between IT development and IT capability - on the other though, it is disturbing to see that they haven't quite merged yet in the data industry.

Why is this important? Well, because when viewed out of context (of each other) the value proposition for these innovations diminishes significantly. Data Architecture, BI Methodology and the expectations for how users will exploit data are part of the same problem space...


Copyright 2013, Stephen Lahanas

Read More

What is Aeronautical Information Management ?

Very few of us who fly from one destination to another have any idea about the information systems that support civil aviation. As one might imagine, the role of technology in helping to manage and regulate the exploitation of our airspace has been growing steadily for decades. Yet, at the heart of every nation’s civil aviation system is something called AIM, Aeronautical Information Management. AIM is the data layer foundation for all other aviation activities and as of today much of it is still manual and un-integrated. The data layer is the foundation for all other modernization initiatives and making sure that it is done right will have a direct impact on the airline industry as well as on passenger safety.

Core AIM Concept

Data as a Product versus Data as a Service – that’s the way that the Global AIM community often portrays the Aeronautical Information Management transformation challenge. Oddly enough though, AIM is the successor to something called AIS – Aeronautical Information Services. The reason for the apparent discrepancy in evolutionary descriptions has to do with a change in how these systems are viewed. The term AIS was derived from ICAO requirements dating back several decades. At that time the provision of manuals extracted from stove-piped data systems was considered a service.

Now of course, we see the term ‘Service’ more closely aligned to specific architectural constructs, i.e. Services Oriented Architecture (SOA). The AIS systems and framework were in fact designed only to support the regular release and distribution of civil aviation knowledge products, which at first were only manifested by printed manuals and later by electronic document distribution. The primary publication is referred to as the Aeronautical Information Publication (AIP) and is released every 57 days.

Aeronautical Information Management is a global initiative and at its core recognizes that in order to move forward to a true ‘Services’ paradigm, civil aviation must consolidate a fractured, stove-piped data layer that had been developed to support paper products and transform it into a single, logical architecture framework.

AIM is part of a much larger global modernization effort for Civil Aviation

AIM Definitions

•    The NAS – A conglomeration of FAA systems which manage the National Air Space.
•    ATM – “Air Traffic Management” is roughly equivalent to the FAA’s NAS.
•    NextGen – The collective name for the set of modernization initiatives which will be rolled out by the FAA over the next decade. 
•    AIXM – The Aeronautical Information Exchange Model. A canonical XML-based data model meant to characterize the entire problem space of civil aviation and provide data exchange support between civil aviation systems through and across AIM infrastructures.
•    SESAR – The ‘Single European Sky ATM Research’ is the equivalent of the FAA’s NextGen efforts.
•    CDM – Collaborative Decision Making (currently mostly manual).
•    SWIM – System Wide Information Management (basically, the support infrastructure for NextGen / SESAR services).
•    NOTAMs – ‘Notice to Airmen’ represents near-real time environmental or situational updates regarding certain portions of airspace.
•    ADS-B - Automatic Dependent Surveillance-Broadcast
•    OATA – The Eurocontrol Overall Target Architecture Activity

Copyright 2012  - Technovation Talks, Semantech Inc.

HyperSmash

Read More

Understanding Master Data Management

I was speaking with recently on the topic of MDM - it occurred to me not too long after we began the conversation that we more than likely had differing perspectives as to what Master Data Management meant. That's inspired me to write this post to talk a little bit about how better to understand MDM.

MDM, The Core Concept:
Let's start with what Master Data is not, Master data is not:

1. Meta-data, which is a description  of data  (or data about data as it's commonly referred to as).
2. Ontology,  Taxonomy or Vocabulary - Master data can be derived from these but is not in itself a formal semantic construct.
3. Software Tool - ultimately, Master Data is technology-agnostic; it is a logical construct which can be defined through various modeling tools and realized through a variety of data management software solutions.  At the point where Master Data becomes tightly coupled with any one software tool or any one modeling technique it will likely loose a great deal of its potential value to the enterprise.

So, then what is it? How would we characterize what can become Master Data or not ?

1. It may be considered "data of record" or an authoritative data source, but it might not be also. Data of record implies that there is a system of record with sanctioned data elements that are not meant to be repeated throughout the enterprise across other systems. Or this might refer to data entities which are determined to be unique and authoritative across the enterprise regardless of their current use (in a system).
2. Master data is reference data, sort of. If we consider that reference data is a definitive set of element definitions or entities associated with any particular business domain, sub-domain or problem space. In this capacity, Master Data may serve multiple roles, including: discovery, registry or repository access, data dictionary foundation.
3. Benchmark - this is a critical consideration; any data entities defined as Master Data elements within an enterprise are unlikely to remain unchanged or unmodified. Eventually there will be variations of Master Data sets, these variations must be tracked back to their source and there must also be a mechanism whereby others in the enterprise can understand where, why and how those modifications occurred 'atop' the core sets of Master Data. Thus the Master Data is a baseline or benchmark wherein the data chain of custody can be managed or tracked. 
4. It can be a canonical data model or data exchange model - this is important in cases where the core data architecture has not yet been designed or deployed, or in cases where it is anticipated that there will be a major or radical transformation of the existing architecture to a new one. The model can contain Master Data elements or sets within it.

MDM in Today's Implementations
Much of what we refer to now as MDM solutions have been borne out of previous product solutions that were describe as meta-data management solutions. For many of the MDM solutions on the market, the "repository or registry" architectural construct / pattern is how this capability is harnessed.  Another architectural approach related to MDM might be referred to as the middleware design - this extends MDM into data transport and is focused on supporting accurate message translation. And of course there are solutions that combine both aspects.

One of the most important aspects of MDM is identifying what constitutes Master and Reference Data

Perhaps we can consider that there are at least two philosophical approaches to MDM:

• Passive MDM - This is most closely aligned to the original Meta-data management solutions with a central repository to support discovery and high level data reconciliation.
• Active MDM - This is most closely aligned with solutions stemming from EAI, Middleware, ETL based solutions where data reconciliation rules are being applied at multiple levels and in more detail.
• Hybrid MDM - Both solutions are relatively weak in dealing with bi-directional reconciliation focused heavily on transactional systems (it is much easier to reconcile historical data from multiple sources than real-time data from multiple sources).  Hybrid MDM applies both previous techniques and new ones to tackle the most problematic use cases.

It is clear to anyone who has worked with database development and data-system integration that having the ability to reconcile data sources adds tremendous value to the enterprise helping to improve performance, integrity and overall efficiency. Being able to do some of this automation using COTS tools is even more appealing, however there is still a set of processes which ultimately takes precedence here if one is to deploy a successful MDM solution. The enterprise data governance approach must be defined first, the data and business environment must be modeled and if ownership of data sets is to be handed off to user groups (either fully or partially) the impact to both governance and model maintenance must be considered and mitigated in advance.

As we've discovered with nearly every IT technology and product over the past 40 years - implementation without process or architectural considerations leads to many issues, often more issues than existed before the technology was introduced.  This is no exception with MDM - the most important thing to consider here is that deployment of MDM software can significantly impact or influence both solution performance and integrity but proceeding without working through the implicit architectural / enterprise issues is risky.

Copyright 2012  - Technovation Talks, Semantech Inc.

Read More