Cyber Security Predictions for 2017

2016 was a big year in the annals of Cyber Security, and 2017 promises to eclipse it.

Creating an Enterprise Data Strategy

An introduction to the process of developing comprehensive strategies for enterprise data manangement and exploitation.

A Framework for Evolutionary Artificial Thought

Let’s start at the beginning – what does this or any such “Framework” buy us?

The Innovation Dilemma

What things actually promote or discourage innovation? We'll examine a few in this post...

Digitial Transformation, Defined

Digitial Transformation is a hot topic in IT and big money maker for consultants - but what does it really mean?.

Showing posts with label Business Intelligence. Show all posts
Showing posts with label Business Intelligence. Show all posts

Sunday, November 10, 2013

Understanding Data Architecture

Someone asked me what at first sounded like a very straightforward question earlier this week; "what is Data Architecture" - or more precisely, what does it mean to you. Usually, I'm not usually at a loss for words when it comes to expounding upon IT Architecture related topics - but it occurred to me at that moment that my previous understanding of what Data Architecture really represents is or has been a little flawed or perhaps just outdated. So I gave a somewhat convoluted and circumspect answer.

Where does Architecture fit within this picture?

The nature of what's occurring in the Data domain within IT is itself changing - very quickly and somewhat radically. The rise of Big Data and proliferation of User Driven discovery tools represents quite a departure from the previous more deterministic view of how data ought to be organized, processed and harvested. So how does all of this effect Data Architecture as a practice within IT (or more specifically within IT Architecture)?

But before we dive into the implications of the current revolution and its subsequent democratizing of data, we need to step back and look again the more traditional definitions as to what Data Architecture represents. I'll start with a high level summary view:

Traditional Data Architecture can be divided into two main focus areas; 1 - the structure of the data itself and 2 - the systems view of whatever components are utilized to exploit the data contained within the systems. Data in itself is the semantic representation or shorthand of the processes, functions or activities that an organization is involved with. Data has traditionally been subdivided (at least for the past several decades) into two categories; transactional and knowledge-based or analytic (OLTP vs. OLAP). 
Now we'll move to a traditional summary definition of Data Architecture practice:

Data Architecture is the practice of managing both the design of data as well as of the systems which house or exploit that data. As such, this practice area revolves around management of data models and architecture models. Unfortunately, the application of Governance within this practice is sporadic and when it does occur is often split into two views: governance of the data (models) and governance of systems (patterns and configurations). 
So, that seems to be fairly comprehensive; but is it? Where does Business Intelligence fit in - is it part of the data management or system management - is it purely knowledge focused or does it also include transactional data? For that matter, do Data Warehouses only concern themselves with analytic data or can they be used to pass through transactional data to other consumers? And isn't Big Data both transactional and analytic in nature? And BTW- how do you model Big Data solutions either from a systems or data modeling standpoint? Now - we start to begin seeing how things can get confusing.

We also need to take into consideration that there has been an attempt made to standardize some of this from an industry perspective - it's referred to as the Data Management Book of Practice or DMBOK. I think in some ways it's been successful in attempting to lay out an industry taxonomy (much like ITIL did) but not as successful in linking that back into the practice of Data Architecture. The following diagram represents an attempt to map the two together...


There isn't a one to mapping between DMBOK and data architecture practice, but it's close
One of the areas that the DMBOK has fallen short is Big Data; my guess is that they will need to rethink their framework once again relatively soon to accommodate what's happening in the real world. In the diagram above, we have a somewhat idealized view in that we've targeted a unified governance approach for both data modeling and data systems.

Let's take a moment and discuss the challenges presented by the advent of new Big Data and BI technology. We'll start with BI - let's say your organization is using Oracle's BI suite - Oracle Business Intelligence Enterprise Edition (OBIEE). Within OBIEE you have a more or less semantic / metadata management tool called Common Enterprise Information Model (CEIM). It produces a file (or files) that maps out the business functionality of all the reports or dashboards associated with the solution. Where does that fit from an architecture standpoint? It has a modeling like interface but it isn't a 3rd normal form model or even a dimensional model. It represents a proprietary Oracle approach (both as an interface and modeling approach). It allows you to track dimensions, data hierarchies and data structures - so it is a viable architecture management tool for BI (at least for OBIEE instantiations). But some traditional Data Architecture groups would not view this as something the architects would manage - it might handed off to OBIEE administrators. This situation is not unique to Oracle of course, it applies to IBM / Cognos and other BI tools as well and there's a whole new class of tools that are completely driven by end users (rather than structured in advance from an IT group).

Now let's look at Big Data. Many of the Big Data tools require command line interface management and programming in order to create or change core data structures. There is no standard modeling approach for Big Data as it encompasses at least 5 different major approaches (as different say as 3NF is from Dimensional). How does an architecture group manage this? Right now, in most cases it's not managed as data architecture but more as data systems architecture. The problem here is obvious; just as organizations have finally gained some insight into the data they own or manage - a giant new elephant as entered the room. How is that new capability going to impact the rest of the enterprise - how can it be managed effectively?

Back to the original question - what is Data Architecture. I'd like to suggest that the practice of Data Architecture is more than the sum of its traditional activities. Data Architecture is the practice of understanding, managing and properly exploiting data in the context of problems any given organization has to solve. It is not limited by prior classifications or practice but has a consistent mandate to be able to represent and hopefully govern in some fashion data as an asset (internally or shared collaboratively). Data Architecture as we know is going to change quite a bit in the next two years and that's a very good thing.



Copyright 2013, Stephen Lahanas



#Semantech
#StephenLahanas

Thursday, October 17, 2013

Revisiting Agile Business Intelligence

The other day the TDWI (the Data Warehouse Institute) sent me a brochure highlighting Agile BI workshops and seminars. Here's how they define it:
"Agile business intelligence addresses a broad need to enable flexibility by accelerating the time it takes to deliver value with BI projects. It can include technology deployment options such as self-service BI, cloud-based BI, and data discovery dashboards that allow users to begin working with data more rapidly and adjust to changing needs.
To transform traditional BI project development to fit dynamic user requirements, many organizations implement formal methodologies that utilize agile software development techniques and tools to accelerate development, testing, and deployment. Ongoing scoping, rapid iterations that deliver working components, evolving requirements, scrum sessions, frequent and thorough testing, and business/development communication are important facets of a formal agile approach. "
Now I found this very interesting given it's something I have been advocating for some time. Although, the definition above left me a bit concerned that what in fact is being suggested is merely the adoption of Agile methodology with minimal regard to Business Intelligence architecture (we've been given a laundry list of related solutions with not clear idea of how they integrate). More importantly, the heavy focus on the development methodology leaves out what we considered the most important aspect of Agile BI (when we first presented this back in 2007) - the end user and how they are integrated into the development process and /or how they drive the very structure of BI by defining it "on the fly" themselves (this goes beyond data discovery).

We presented this in the Fall of 2007 in Chicago
Agile BI must encompass a wider architectural approach...

Since 2007, a number of tools have come out that specifically answer this end-user consideration. A good example of this is Tableau (which markets itself as "visual analytics for everyone"). So on the one hand it is both gratifying and exciting to see that Agile concepts are being extended to Data Architecture and that new products are being introduced to help bridge the gap between IT development and IT capability - on the other though, it is disturbing to see that they haven't quite merged yet in the data industry.

Why is this important? Well, because when viewed out of context (of each other) the value proposition for these innovations diminishes significantly. Data Architecture, BI Methodology and the expectations for how users will exploit data are part of the same problem space...


Copyright 2013, Stephen Lahanas

Monday, November 5, 2012

Artificial versus Natural Intelligence



Too often we dismiss the Semantics of a problem space as a superficial element of the larger whole. However, that is often not the case at all, especially where the Semantics of describing the problem directly interfere with the ability to achieve expectations related to the problem space. Case in point is the field of "Artificial Intelligence" or AI within IT. AI was first describe as a discipline in the late 1950's and has been a part of our popular culture ever since Arthur C. Clarke wrote 2001, A Space Odyssey in 1968. While the question of intelligence in computers or robots had been dealt with previously in popular science fiction; 2001 gave us an entirely new perspective on AI - one that seemed entirely plausible and not too far off from being realized (and hopefully not the part where HAL has a nervous breakdown). HAL or Heuristically programmed Algorithmic computer, was an intelligence that had not been programmed but rather had learned in somewhat the same way humans do; this construct is also closely tied with certain aspects of the computer science discipline of AI and is referred (somewhat inaccurately) to as Machine Learning. Yet, nearly 60 years after AI was envisioned, many feel the field has yet to reach any of its original objectives.

Inside HAL - 2001 A space Odyssey
This is not to say that phenomenal advances in IT have failed to materialize, because we have experienced remarkable gains in a variety of areas. However, the nature of those advances seems somewhat different than what we have been expecting in relation to the question - "what is intelligence." Or perhaps the question more precisely needs to be; what is intelligence in relation to viable Information Technology solutions? Today in IT, we tend to use the term intelligence sparingly; we have one major field associated with analytics called Business Intelligence, but we haven't been willing to go much beyond that in claiming that what we provide exhibits anything we might consider related to the concept of intelligence.

So, let's go back and ask a fundamental question, what is it that we meant by Artificial Intelligence in the first place? Here's the wikipedia definition:

"Artificial intelligence (AI) is the intelligence of machines and the branch of computer science that aims to create it. AI textbooks define the field as "the study and design of intelligent agents"[1] where an intelligent agent is a system that perceives its environment and takes actions that maximize its chances of success.[2] John McCarthy, who coined the term in 1955,[3] defines it as "the science and engineering of making intelligent machines."

Let's contrast this with the definition for the term Intelligence by itself:

"A very general mental capability that, among other things, involves the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly and learn from experience. It is not merely book learning, a narrow academic skill, or test-taking smarts. Rather, it reflects a broader and deeper capability for comprehending our surroundings—"catching on," "making sense" of things, or "figuring out" what to do."

Looking at these definitions, a couple of things become apparent immediately:

  1. That computer science though AI has moved towards fairly narrow definition of Intelligence.
  2. Computer Science has not to date managed to reproduce the vast majority of attributes that we tend to associate with (human or generic) Intelligence.
  3. The two terms being defined, which upon first hearing them may seem fairly similar, are not in fact closely related at all.

That last point is critical because it has everything to do with developing and or achieving expectations. As the expectations within the IT domain for AI have in fact gotten somewhat narrower than those which they began with, the expectations across the general public for what Artificial Intelligence will encompass have remained the same or even expanded. It can be summarized thus; most people when they hear the term AI expect that it will result in some sort of thinking machine and thinking cannot be defined or conceived of by most people in terms different than are applied for characterizing human thought.

Our focus over the past several decades has been directed more towards brute force computing - the ability to jam ever more computations into smaller machines at an ever-expanding rate of speed. While this is no doubt vital to the overall goals of achieving true intelligence it has not led to any serious development to what might be referred to a computational cognition. Cognition is the set of functions related to taking the lower level processing and apply real meaning to that "sensory input" without the aid of outside interpreters (e.g. human beings).

Watson won big on Jeopardy
So this brings us to our premise; are we in fact trying to do two separate things in the context of one field of practice and research when we should be splitting it in two? If we consider that there may be at least two separate types of intelligence that can be achieved in computer science, one Artificial in nature and one Natural, might we not be able to realign efforts and expectations accordingly?

This would need to start with some definitions, so we'll return to the Semantics. What if we retain the narrow view currently associated with AI (which is focused on statistical theory, agent technology etc.) as it exists now and consider that Artificial Intelligence is any technology or resulting intelligence that aids, facilitates or other empowers human cognition. In other words, Artificial Intelligence becomes a super-charged version of Decision Support. Then we could pursue a separate (albeit potentially related) field of endeavor that can be referred to as Natural Intelligence. We call it Natural because in fact what we're after is something that functions or behaves in a more human way - something specifically designed to mimic human cognition. 

This year we witnessed Watson defeat several Jeopardy champions; however it is not likely even with that impressive result that we could characterize how Watson operates as Cognition. However, what IBM is trying to do now with Watson is much more akin to Natural Intelligence than what is considered to be traditional AI - even if some of the same technology is being applied. The difference is one of goals and objectives - the expectations. IBM's expectation was the Watson had to parallel human cognition in a typically human task (closer to real time as opposed to earlier efforts with Chess) and it represents perhaps the first real step in achieving Natural Intelligence and realizing the larger set of AI goals first described in 1950's. The larger challenge now is to stop confusing these fields with one another and direct the necessary efforts towards improving the outcomes associated with each. We will discuss how that might be accomplished in future posts...


Copyright 2012  - Technovation Talks, Semantech Inc.

Friday, November 2, 2012

Understanding Data Warehouse Challenges

Last week we talked about some of the issues arising from the emerging field of Big Data, today we're going to explore a more mature data solution and point out some of the challenges that it's faced - and some of those challenges were instrumental in pushing the IT industry towards Big Data.

The Data Warehouse concept is built atop the notion that all data related to the enterprise can be captured and centrally or holistically managed. This is a powerful idea, yet there is more than one way to achieve that goal. The traditional view of the EDW attacked the problem from a very DBMS-centric perspective. This is primarily why EDW projects become so expensive, difficult and ultimately hard to adopt. The typical EDW approach attempted to gather all of the data related to the enterprise and place it into one massive repository structure. Whether this approach was attempted in chunks or as a “Big Bang” assault made little difference in the long the run as the byproducts of the practice were the same; those byproducts included:
  • A more bureaucratic management approach to the data layer in general.
  • An added degree of separation between the data owners and the data developers.
  • A certain level of inflexibility in regards to how data was updated, corrected or otherwise transformed.
  • An added degree of separation between database developers and data exploitation developers.
  • An added degree of separation between database developers and application developers.
  • An inability to quickly respond to major changes in the business.
  • Dependence upon a sub-set of industry experts and equipment that is more expensive than the industry norm.
  • A higher cost associated with scalability in general.
It is worth examining some of the core concepts associated with Data Warehousing a little bit closer to help understand why these outcomes tend to result from the traditional EDW approach.

DBMS Focus – At the time when EDWs became popular, other areas of data architecture were only just beginning to blossom. Today’s Business Intelligence platforms represent much more than mere reporting engines. Metadata management was only just beginning to be understood in the mid-1990’s and focus on Semantic technologies was virtually non-existent. The world according to DBMS in 1995 had a relational management system in the middle with ETL feeding into and reports coming out. This might be thought of as a three layer, stove-piped database systems view of the data architecture.

The Enterprise Single Instance – While consolidating like capabilities into marts or stores or some other ‘functional single instance’ approach has achieved quite a bit of success over the past two decades, attempting to manage all data in one structure has proven much more difficult. This is why the notion of Massively Parallel Processing (MPP) was needed to make it viable back in the 1990s. MPP in the context of proprietary hardware was expensive though and perhaps failed to recognize the power of networked processors on inexpensive hardware (i.e the Google scalability model). The other key consideration here was the added steps that were needed in order to make such a system perform within reasonable parameters. So, the single instance enterprise faced and still faces major hurdles in terms of costs, manageability and performance. 


Data is no longer confined within the context of single systems
EDW Fallacies
If we were to directly challenge the core EDW assumptions and illustrate the fallacies associated with the philosophy, our list would resemble the following:
  • The Business will remain static over a relatively long period of time.
  • The Enterprise will remain static over a relatively long period of time.
  • That source data and data exploitation should not be managed synergistically, in other words that Decision Support or Business Intelligence solutions built on top of EDW source data should be viewed as separate, albeit related efforts.
  • That the data layer and the application layer can or should be viewed or designed separately.
  • That computer Hardware would not catch up to the processing load – i.e. that the data layer would always require specialized Massively Parallel Processing (MPP) in order to manage very large quantities of data. Furthermore, this assumption also implied the data would remain in a single instance data source. So instead of parallel processors deployed in specialized equipment, Big Data now uses the cheapest processes / equipment possible in a commodity approach with data spread out in sets of distributed file systems. In fact this has been take even further as this week the US Government announced the completion of the world's most powerful supercomputer. It achieved all of its latest gains by using commodity hardware (in this case, GPUs, game video processors widely available on the market).
  • That network architecture, data architecture, application / SOA architecture and enterprise architecture are separate.
  • That the Internet (Cloud) would not represent a viable mechanism for connecting to distributed data sources.
  • That unstructured data was not as valid as structured data (mainly because no mechanism existed to incorporate into the traditional database management approaches).
  • That most major transformations need to occur before data is placed into the primary storage / management entity (i.e. DBMS, warehouse).
  • That there is a single version of the truth, period. This is perhaps the biggest fallacy behind all data warehouse, MDM and Governance solutions. Data can be managed, but it is dynamic and all always will be. Viewing data as incontrovertible, orthodox truth immediately eliminates much of the value that data otherwise provides. Situations change, and every stakeholder views the whole from their unique perspectives. Yet, there can still be order in a relativistic environment (much as there is in the real world).  This doesn't mean data cannot be standardized or managed - it merely takes into consideration the inevitable evolution that will occur.


Copyright 2012  - Technovation Talks, Semantech Inc.