Cyber Security Predictions for 2017

2016 was a big year in the annals of Cyber Security, and 2017 promises to eclipse it.

Creating an Enterprise Data Strategy

An introduction to the process of developing comprehensive strategies for enterprise data manangement and exploitation.

A Framework for Evolutionary Artificial Thought

Let’s start at the beginning – what does this or any such “Framework” buy us?

The Innovation Dilemma

What things actually promote or discourage innovation? We'll examine a few in this post...

Digitial Transformation, Defined

Digitial Transformation is a hot topic in IT and big money maker for consultants - but what does it really mean?.

Showing posts with label Big Data. Show all posts
Showing posts with label Big Data. Show all posts

Sunday, January 1, 2017

Whatever Happened to Knowledge Management?

Information Technology is a dynamic field, one often driven by buzzwords and fleeting trends. Sometimes these trends continue for decades, other times they fade somewhat quickly. One trend that experienced that fate seems to be Knowledge Management (KM). I recall first hearing about it in the early 2000’s and at that time it seemed to encompass several classes of subordinate technologies including but not limited to the following:
  • Document Management
  • Content Management
  • Search technology (various)
  • Business Intelligence (A.K.A. Decision Support or Analytics)
  • Metadata Management
  • Learning Management

It also tended to include more specialized tools such as knowledgebases or FAQ generators and it potentially seemed to include more integrative content-focused technologies such as Wikis. For a time KM seemed poised to also include a wide range of Semantic technologies as well. However, in the last two years or so in particular, the term Knowledge Management has seemingly dropped off of the map. I was wondering to myself why this may have happened and whether anything in particular had replaced it.
To be sure, there are a couple of related trends that have dominated the IT landscape in the past two or three years; the most significant of those being Data Science & Big Data. However, neither of these seems to fulfill the role KM was being groomed for over the previous decade. In fact, in some ways these more recent trends have become much less specific in regards to their expectations or scope (which has actually become a problem for both of them). This situation may actually help to explain what happened to Knowledge Management - perhaps the original scope was too expansive? But was it just a scope issue?
I want return again to the core or implied premise associated with Knowledge Management, that there ought to be some sort of enterprise-wide ability to help unify all of these knowledge related functions or processes and resources. The problem with this premise is related to the scope, in that there isn’t one product group of technologies associated with it, but rather a fairly large set of technologies – some of them not closely related at all other than in a philosophical sense – in other words that they could be construed as part of a larger knowledge ecosystem. So, we seem to be missing an industry impetus, but we also seem to be lacking any sort of agreed upon knowledge process or framework that would necessarily help to tie all of these diverse technologies together. This latter problem takes us deep into the heart of the larger philosophical question which KM seemed to be begging – e.g., what is the difference between information and knowledge? That isn’t an easy question to answer – and in some sense parallels my recent discussion on Artificial Intelligence versus Artificial Thought. In fact, AI could even be considered as part of KM depending on how you look at it.
So back to the tough question, what makes data or information become knowledge? Is that dependent on adding value to the data or information though specific types of processes or is it merely in the integration and analysis of such source data that the source transcends itself to become knowledge? Or is knowledge only something we can consider in a collective sense; with the sum total of all data assets being knowledge potential of some sort? These types of questions may have been raised during the years that KM was discussed actively but I think never properly answered.
You might have noticed from my initial list of related technologies that I seemed to have left out Data Management or Information Management. Either of those could potentially be considered to be part of a knowledge management framework – however the reason I left them out is that for those fields, there is a much better operational understanding of how those type of technologies work. In fact, the view from with one of these areas, Data Management overlaps quite a bit with some of what I’ve attributed to Knowledge Management (one need only look at the DMBOK to see this illustrated). Thus Data Management as a trend has continued (for decades now), primarily concerned with operational maintenance of a number of unrelated technologies without the implied necessity to integrate it all into something transcendent across the enterprise.
What if we did wish to settle the deeper question though, regarding the differing expectations between operational management of source data and knowledge exploitation? It is now 2017, are we in a position to define an integrative knowledge framework and if so what would the philosophical foundation consist of? Moreover, would coming up with this type of framework help to redeem the dimming trends of Big Data and Data Science? I think it’s worth trying to answer the question and also worth taking a shot at defining the missing knowledge framework. I will tackle both parts of this problem in two upcoming posts…
Copyright 2016, Stephen Lahanas

Friday, August 8, 2014

Building Effective IT Strategy - part 3

In our last two post on IT Strategy, we highlighted how Strategy can be structured, how it differs fro Tactics (although we will explore that in more depth in this post) and how to employ a consistent process towards developing strategy. The first two steps involved determining what the core strategic approach might be; the second focused on goal-setting. The third and most difficult part is assigning actions to goals...

So, using our Big Data case study, how would begin to translate the higher level goals into definitive actions and then what types of tactics might be used to carry out those actions?


This illustration highlights where Big Data fits within a larger set of Strategic elements in an overall Transformation initiative. This type of representation helps to define relationships, dependencies and quantifies where work needs to happen once the higher level goal-setting has been defined.  

The types of actions that might be involved with actualizing a Big Data Strategy might include the following:

  • Creation of a team or center of excellence to manage the technology / project
  • Definition and Deployment of a proof of concept 
  • Acquisition of the raw data intended for use in the Big Data solution (so let's say this is for an energy company it might include SmartGrid sensor information).
  • Acquisition and / or development of the Big Data Platform
All of these possible actions of course imply a number of key decisions that must be made; the following are a few examples of those;
  • Determination of Big Data technology to use (triple store, key value etc.)
  • Determination / selection of a Big Data solution hardware platform
  • Determination of modeling or data profiling approach
  • Choice of BI platform for data visualization

All of this information is going to be necessary in order to complete detailed roadmaps and ensure accurate estimates for those who manage the IT portfolio process in any given organization. Actions can then begin to be translated into milestones with traceable costs. Those Action-Milestones are then mapped specifically to goals/objectives previously identified in the higher level strategy.

Now, how does action to goal alignment involve Tactics? In the case we've introduced and in most others, the tactics involve the core tools for decision-making. So, for all of the decisions listed above, individual analyses of alternatives might be conducted. For product decisions, run-offs / competitions / evaluations and source selection processes are applied. For design considerations, an architecture approach is applied. All of these activities can also fit within a lifecycle process - all of this represents Tactics.  Why? because, we could use roughly the same lifecycle approaches for any type of technology - whether it is UAV development, Quantum Computing or building a SharePoint portal. It is the interchangeable actualization toolset for all strategy.

The hardest part of aligning Strategy, sub-strategies and tactics is when you find yourself in a very large transformation effort (one perhaps dealing with 100's of systems, dozens of technologies and perhaps thousands of people). There is no single solution, tool or approach for managing that - it represents what mathematicians often refer to as a unsolvable problem (NP Hard). We will look at IT Transformation and intense complexity in an upcoming post.


Copyright 2014, Stephen Lahanas


#Semantech
#StephenLahanas
#TechnovationTalks

Thursday, August 7, 2014

Building Effective IT Strategy - part 2

In yesterday's post, I introduced three elementary categories for IT Strategy:

  1. Product 
  2. Portfolio
  3. Transformation

I then pointed out that each of these follows a basic cycle of:

  1. Determination of Strategic Approach
  2. Goal-Setting
  3. Assigning actions to goals

So, now as promised, I will use this to take a look at a specific scenario. But before we begin, let me mention an important caveat to the three categories listed above. While all strategy tends to fit within these, not all strategy has to exist at the highest level. In other words, there are various levels of Strategy that are still abstract enough to remain differentiated from Tactics.

In this case study example, we are going to look at Big Data. Big Data is something that many organizations might consider important enough and large enough to develop a strategy around. However, just looking at the moniker "Big Data" and one might instantly wonder - well, doesn't that belong as part of a larger "Data Strategy." Yes. And then wouldn't that also imply that the Data Strategy would be part of a larger Strategy. Again the answer is yes - and in this case the whole thing lines up neatly like this:

  • Portfolio Strategy
    •  Data Strategy
      • Big Data Strategy

So, this begs the questions, just how we might first differentiate the lower level strategy from higher level strategy and then perhaps even more importantly, how do you ensure they stay in alignment?

Differentiation:
This tends to be managed something like this - you begin at the top level with the superstructure of where everything is supposed to fit as well as common capability / design principles and objectives. Then as you move down the Strategy levels, things progress from conceptual expectations to logical descriptions. The top level is the most open or flexible; the bottom level the closest to expectations regarding solution execution.

Reconciliation:
Side by side with the differentiation activity is integrated road-mapping - each level below fitting neatly into the one above. The other big component here is alignment of Strategy with Architecture which provides the other key reconciliation tool (if used properly).

So, how would a Big Data Strategy fit into an Enterprise Data Strategy? First, it obviously in most cases extends something that already exists. This then implies either replacement of existing capability or addition of new ones. Now let's jump back to the process we mentioned earlier - Step 1 - assigning portfolio strategy is complete. How would we attack goal-setting for Big Data?

This by the way, is where perhaps more than half of the organizations trying to adopt Big Data solutions are getting tripped up right now. A poor example of goal-setting would be - "let's do a POC without a clear path of how to exploit this technology yet" (mainly because everyone else seems to be doing it. A better approach might be:

  1. Define the set of possible Use Cases associated with your organization (where Big Data might make an impact)
  2. Choose one or two that can be effectively demonstrated and measured - let's say one might be the rapid development of a user driven BI solution based on unstructured (web/social media) data. 
  3. Develop a clear path as to how; a) the initial capability could be rolled into the larger existing ecosystem to avoid silos or solution-fracking and b) add new functionality to the emerging Big Data solution - consistent with overarching organizational goals. 

Step 3 is assigning actions to goals. We'll take a look at that in our next post...




Copyright 2014, Stephen Lahanas


#Semantech
#StephenLahanas
#TechnovationTalks

Tuesday, August 5, 2014

What is Digital Transformation?

It sounds a bit misleading perhaps - maybe one gets the impression that this is something akin to rendering special effects for movies. In mainstream IT however, Digital Transformation has come to mean something quite different.

The shorthand definition for it is this; "the coordinated integration of all customer-facing digital capabilities." This includes mobile, web and social media. There is perhaps an implied level of underlying integration supporting of all it, but that isn't always required - at least not at first anyway.

We use the word "customer" here a bit loosely, as Customer can refer to any users or members of a certain organization. So, a government entity might consider citizens as customers and the military could consider its rank and file 'customers' in this sense also. Digital Transformation can occur in just about any type of organization specifically because the set of technologies involved is common across industries.

Ten years ago a 'Digital Transformation' would have likely been referred to as a "Portal Strategy." At the time, it was thought that the portal metaphor (and associated software products) would suffice to meet all customer-facing needs. A typical portal might include the following elements in 2005:

  • Extranet
  • Intranet
  • Collaboration tools
  • Content Management tools

What happened since then? Well, at least two and possibly three revolutions in technologies with more on the cusp; the key revolutions include:

  • Mobile - (not laptops of course, but the smartphone and mobile)
  • Social Media - some re-branding of collaborative capabilities merged onto smartphones as well
  • Cloud Computing - re-branding of visualization - and importantly opening that up to customer facing apps (web and mobile).
  • with Big Data coming on strong (although to be honest, most places still don't know how to use it so it's an outlyer with respect to customer facing technology).

On the backend of course are all the systems that actually make the organization go - but for the customer this is like the engine within the chassis. Most customers don't care what's under the hood until or unless it fails to perform.

Digital Transformation is a lot like enterprise integration but on the customer facing side of things. It is the alignment, coordination and update in tandem of the core mission apps and interfaces. And it can potentially include the Web of Things as well. So for example, if you have a grocery store then perhaps there are kiosks inside and gas pumps outside and nifty new cooler video displays etc. All of that technology is customer-facing so for a true transformation it all must be managed within a shared context.  The transformation has one goal - to go from a heterogeneous mix of random technologies to a holistic solution designed to facilitate strategic goals.

So, how does an organization undergo a Digital Transformation? We will look at this in more detail in one of our upcoming posts.


Some organizations (like larger retailers) face dual Digital Transformations; in so much as they must serve both traditional customers and the needs of many thousands of employees (with separate capabilities) such as in the example above targeted to employees...


Copyright 2014, Stephen Lahanas


#Semantech
#StephenLahanas
#TechnovationTalks

Friday, December 13, 2013

The Challenges to Global Collaboration

There has been a lot of discussion lately on the web about how we're beginning to achieve the decades old goal of consistent global collaboration for innovation and problem-solving.  The celebration may be a bit premature, not unlike prior victory dances we engaged in related to:

  • Cancer Research
  • Flying Cars or Electric Cars for that matter
  • Space travel
  • AIDs Cures
  • Big Data changing the world
  • Artificial Intelligence and so on...
We have had a bad habit lately of confusing the initiation of a trend or innovation with its mature realization. The technical foundation for global collaboration has been developed and deployed over nearly four decades. That's how long it has taken to define the network communications paradigm and deploy the necessary bandwidth, information resources and devices to make such a lofty goal possible. But then again, perhaps we need to step back and ask ourselves what Global Collaboration really is and identify the remaining barriers that may otherwise hold it back.



Global Collaboration represents the exploitation of both technological assets and cultural predispositions to share knowledge and jointly resolve common challenges. Now, this model isn't entirely new is it? Academia has been doing this for centuries - sort of. Within higher education there is (and always has been) a tacit expectation for various types of cross-institutional mind-share. However that expectation was always restricted by the following factors:
  1. Competition for ownership of ideas.
  2. Limits in the ability to communicate (technology).
  3. Limits in access to the right information (technology and cultural boundaries or secrecy - and this factor is of course related to factor 1).  
  4. A limited working model in how to organize "virtual communities." In the old days, these communities were characterized as scientific societies and much of what they accomplished was based on direct point to point correspondence, journals and meetings. We are still working with all of those metaphors today even though we have witnessed the birth of real-time communities powered by the combination of social media and mobile devices. 
  5. Orthodoxy. This may sound odd, but in fact it is orthodoxy that puts the institution into "institutionalism." In other words, it represents the ultimate barrier to acceptance of or even discussion of unorthodox or disruptive concepts. 

As much as we'd like to think that modern society is open-minded and innovative - and that we're hurtling from one innovation to another at breakneck speed - that's just not the case. The telephone was invented in 1880's, radio communication was invented before 1920, television around 1930 and computers in the 1940's, miniature transistors in the 1950's  - yet the journey from those various inventions to a mobile device that combines them took more than 60 years. The story of the electric car is much worse - it was invented at roughly the same time as the internal combustion approach was worked out - yet one was promoted and the other neglected. We don't typically move nearly as fast as we think we do and we often make poor choices along the way.

What does this have to do with Global Collaboration? Well, the premise goes that if dozens or hundreds or even thousands of minds were directed at a problem then the solutions to that problem would happen faster and be vetted better. It would no longer be Edison versus Telsa, but countless inventors competing and collaborating on a level playing field - or at least that's idea. The closest thing to this model that we have now is Open Source software.

But does the Open Source software model represent the type of Global Collaboration that futurists have been predicting for the past half century?  The answer is yes and no.

Yes, it represents a prototype of highly specialized working collaboration (on a global scale) - no it doesn't seem to represent the prototype for making truly revolutionary breakthroughs. So, why isn't the "Open Source" model redefining innovation and progress in software or science? Here are some possible explanations:

  1. The (majority of) projects are too narrowly defined to make real breakthroughs.
  2. The context (software development) is too limited. 
  3. Coding is focused on skill in execution - and problem solving at a tactical scale. It doesn't usually require any quantum leaps in conceptual understanding or practical application.
If you think I'm being harsh about the Open Source movement as a force for global innovation - then ask yourselves how much of the core technology that is powering our current infrastructure came out of it:
  • Well, they gave us MySQL - and SQL came from previous working groups - but it was also very similar to existing products.
  • What about Java, Linux, Apache etc. All good stuff, but not truly disruptive - the seeds for all of that had been developed elsewhere. 
  • Big Data, Cloud tech - again the same above.  

Lot's of good stuff is coming out of the Open Source movement - it's just not that revolutionary and much of what has been revolutionary has come out of the older style collaborative ecosystems like Academia and DoD (or combinations thereof).

So, now with all of that as preface; here is a list of some of the challenges that I think is holding back global collaboration (at least the way it has been envisioned):

  1. Competition can be both beneficial and destructive. We need to find a better way to manage it and ensure open playing fields for smaller players. In other words, how can we provide tangible rewards for contribution that doesn't end up excluding the majority of participants? 
  2. Secrecy -  One of the lasting holdovers of the Cold War is a subculture of secrecy that seems to be present in almost every major nation on the planet. This represents perhaps the single biggest obstacle to the eventual global cooperation that futurists tend to describe. Solving problems together requires a level of trust that simply doesn't exist yet. Hopefully it will someday.
  3. Orthodoxy - How many times have brilliant ideas been dismissed because the community in which they were introduced rejected them? More times than can be counted no doubt. The biggest barrier to most innovation is and always has been a lack of imagination. When careers depend of defending existing paradigms; newer paradigms will take longer to birth or die in the cradle.
  4. A framework or methodology for global problem solving. There's perhaps thousands of these or similar approaches floating around (from academia to open source software for example) but a truly effective one remains elusive. 

My next post will explore what "a truly effective methodology for global problem solving" might look like.




Copyright 2013, Stephen Lahanas

#StephenLahanas
#TechnovationTalks
#SemantechInc.

Sunday, November 10, 2013

Understanding Data Architecture

Someone asked me what at first sounded like a very straightforward question earlier this week; "what is Data Architecture" - or more precisely, what does it mean to you. Usually, I'm not usually at a loss for words when it comes to expounding upon IT Architecture related topics - but it occurred to me at that moment that my previous understanding of what Data Architecture really represents is or has been a little flawed or perhaps just outdated. So I gave a somewhat convoluted and circumspect answer.

Where does Architecture fit within this picture?

The nature of what's occurring in the Data domain within IT is itself changing - very quickly and somewhat radically. The rise of Big Data and proliferation of User Driven discovery tools represents quite a departure from the previous more deterministic view of how data ought to be organized, processed and harvested. So how does all of this effect Data Architecture as a practice within IT (or more specifically within IT Architecture)?

But before we dive into the implications of the current revolution and its subsequent democratizing of data, we need to step back and look again the more traditional definitions as to what Data Architecture represents. I'll start with a high level summary view:

Traditional Data Architecture can be divided into two main focus areas; 1 - the structure of the data itself and 2 - the systems view of whatever components are utilized to exploit the data contained within the systems. Data in itself is the semantic representation or shorthand of the processes, functions or activities that an organization is involved with. Data has traditionally been subdivided (at least for the past several decades) into two categories; transactional and knowledge-based or analytic (OLTP vs. OLAP). 
Now we'll move to a traditional summary definition of Data Architecture practice:

Data Architecture is the practice of managing both the design of data as well as of the systems which house or exploit that data. As such, this practice area revolves around management of data models and architecture models. Unfortunately, the application of Governance within this practice is sporadic and when it does occur is often split into two views: governance of the data (models) and governance of systems (patterns and configurations). 
So, that seems to be fairly comprehensive; but is it? Where does Business Intelligence fit in - is it part of the data management or system management - is it purely knowledge focused or does it also include transactional data? For that matter, do Data Warehouses only concern themselves with analytic data or can they be used to pass through transactional data to other consumers? And isn't Big Data both transactional and analytic in nature? And BTW- how do you model Big Data solutions either from a systems or data modeling standpoint? Now - we start to begin seeing how things can get confusing.

We also need to take into consideration that there has been an attempt made to standardize some of this from an industry perspective - it's referred to as the Data Management Book of Practice or DMBOK. I think in some ways it's been successful in attempting to lay out an industry taxonomy (much like ITIL did) but not as successful in linking that back into the practice of Data Architecture. The following diagram represents an attempt to map the two together...


There isn't a one to mapping between DMBOK and data architecture practice, but it's close
One of the areas that the DMBOK has fallen short is Big Data; my guess is that they will need to rethink their framework once again relatively soon to accommodate what's happening in the real world. In the diagram above, we have a somewhat idealized view in that we've targeted a unified governance approach for both data modeling and data systems.

Let's take a moment and discuss the challenges presented by the advent of new Big Data and BI technology. We'll start with BI - let's say your organization is using Oracle's BI suite - Oracle Business Intelligence Enterprise Edition (OBIEE). Within OBIEE you have a more or less semantic / metadata management tool called Common Enterprise Information Model (CEIM). It produces a file (or files) that maps out the business functionality of all the reports or dashboards associated with the solution. Where does that fit from an architecture standpoint? It has a modeling like interface but it isn't a 3rd normal form model or even a dimensional model. It represents a proprietary Oracle approach (both as an interface and modeling approach). It allows you to track dimensions, data hierarchies and data structures - so it is a viable architecture management tool for BI (at least for OBIEE instantiations). But some traditional Data Architecture groups would not view this as something the architects would manage - it might handed off to OBIEE administrators. This situation is not unique to Oracle of course, it applies to IBM / Cognos and other BI tools as well and there's a whole new class of tools that are completely driven by end users (rather than structured in advance from an IT group).

Now let's look at Big Data. Many of the Big Data tools require command line interface management and programming in order to create or change core data structures. There is no standard modeling approach for Big Data as it encompasses at least 5 different major approaches (as different say as 3NF is from Dimensional). How does an architecture group manage this? Right now, in most cases it's not managed as data architecture but more as data systems architecture. The problem here is obvious; just as organizations have finally gained some insight into the data they own or manage - a giant new elephant as entered the room. How is that new capability going to impact the rest of the enterprise - how can it be managed effectively?

Back to the original question - what is Data Architecture. I'd like to suggest that the practice of Data Architecture is more than the sum of its traditional activities. Data Architecture is the practice of understanding, managing and properly exploiting data in the context of problems any given organization has to solve. It is not limited by prior classifications or practice but has a consistent mandate to be able to represent and hopefully govern in some fashion data as an asset (internally or shared collaboratively). Data Architecture as we know is going to change quite a bit in the next two years and that's a very good thing.



Copyright 2013, Stephen Lahanas



#Semantech
#StephenLahanas

Sunday, September 22, 2013

Top 5 Reasons IT Projects Fail

Last Fall, I wrote an article for Dice.com about the top 5 reasons ERP projects fail: that was in response to news of a specific project shutdown. However, the topic as to why IT projects in general seem to be so failure prone is worth exploring as well.

As far as hard statistics as to how many IT projects succeed or fail; it's difficult to get a clear picture. Part of the reason this is the case is that often when a project is clearly going to fail it gets "redefined" rather than cancelled. That redefinition represents a re-baselining effort that dramatically scales back expectations with the hopes that the reduced scope can be met. Sometimes that works, other times re-baselined projects are later cancelled anyway. If our measure for success was the original set of expectations associated with a project, the likely rate of IT failure would be extremely high - perhaps in excess of 50% of all IT projects.

In IT, building bridges to nowhere is fairly common, but no less costly...
This is a rather incredible statement of course, especially if we compare IT with most other types of projects across industries. What exactly does this level of failure really indicate though? It could represent any or all of the following factors:

  • That the expectations of the stakeholders were out of touch with the reality of what their IT provider organizations could manage.
  • The level of complexity was not properly gauged or managed.
  • IT is changing so fast now that a project defined last year might require major changes if it is still 'in motion' this year. 
  • The project was driven more by the perception of some need than the reality of a need. This of course refers to all those projects that are spun up in response to industry hype without a clear set of requirements. 
  • The technology assumptions (driven from the expectations) were hastily made and then never challenged later (or perhaps more appropriately - no challenge was permitted). 

While these five factors might be taken as the reasons that so many IT projects fail, they are in fact only the backdrop for the real reasons. Our focus needs to be directed to the fifth bullet point above - assumptions. Thus the five reasons we're focusing on here are related to why failure occurs after a project has been launched.

Even if a project wasn't launched under the best of circumstances, it still has a chance of succeeding if it is flexible enough. But what makes a project flexible? Is it the application of some Agile methodology or some other unique lifecycle management approach? Maybe not. Let's look at the real top five reasons that IT projects fail once they've begun:

  1. An inability to examine or challenge assumptions.
  2. A "silo" mentality.
  3. The inability to compromise.
  4. Technology-centric rather than capability centric perspective.
  5. Poor role definition and lack of project dynamics.

Let's look at these a little closer now...

1 - As noted previously, the inability or unwillingness to challenge assumptions is a huge issue on many projects. This is a problem precisely because of the manner that many projects get spun up (the factors listed previously). This is a little different than the traditional dialog on 'requirements' that's heard when explaining Agile methodology because in that conversation the focus is on the detailed design. Here the focus is on the big picture assumptions which in truth are much harder to face or deal with.  In any project, there are perhaps less than 10 issues which will determine the success or failure of the effort. You may have another 10,000 detailed issues or requirements in a backlog and each of those may be dealt with successfully yet still not save the project from failure if the core issues aren't addressed. Those core 10 or so issues are almost always associated with the initial assumptions connected to the project.

Those types of issues include things like the original intent of the solution, preliminary technology choices and solution architecture decisions as well as organizational approaches. There is no one correct mix of all these types decisions that ensures a successful project and that's precisely why those decisions should be reexamined on a regular basis. In other words, we shouldn't view the first hectic weeks or months of a larger initiative as infallible exercises in perfect thought or planning, but for some reason, many people do.

2 - The Silo Mentality. This refers to projects that become over-compartmentalized; where each sub-unit views itself more or less divorced or separate from every other. While everyone on a project needs to understand their roles (point 5) by the same token they must also all share responsibility for the outcome. There is nothing sadder and less justified than internal finger-pointing when failure occurs. This is a problem that is easily corrected with the right leadership and project structure. Whenever you begin hearing various players within a large project stating that issues relating to them are someone else's concern or responsibility then you know the project has developed this problem.

3 - The Inability to Compromise - We talked about this in a previous post from the perspective of team leadership; here compromise has an even broader context. Compromise here represents compromise between teams, by leaders - as well as compromise on large, tough issues. Lack of compromise is one of the chief symptoms of an 'inflexible' project. People will always have excuses and cite deadlines yet invariably where compromise fails to occur, deadlines are shot anyway and often the entire project ends up being cancelled.

4 - Technology-Centric Projects - This may sound funny coming from an IT geek however it is nonetheless valid. Let's look at Big Data as an example. What exactly does that new technology category mean to a business that wishes to jump onto the hype bandwagon? Does it imply faster data systems, better storage management or is it tied to specific Use Cases associated with that organization? If the expectations end up looking just like a Gartner article on who's doing what instead of Use Cases that illustrate how it can be exploited in your organization then you might say you've got a technology-centric project. The fact that you think everyone else is doing something shouldn't drive your decisions. If you've been stuck with a hype driven project then use the initial period to re-tune it to ensure that its aligned to your business. If you find that it's not then have the courage to reverse course if necessary (the earlier, the better - you can always come back to it later).

5 - Poor Role Definition and Project Dynamics - How you build the project is usually just as important as the underlying assumptions behind it. Understanding roles and how those roles and groups ought to work with one another is important. For example, an architect and a tech lead are two very different roles - one owns the technology the other designs the solution (completely different perspectives). Yet many organizations regularly confuse those two roles. Also, there's often not clarity in regards to who has oversight or leadership across groups. This last point relates 'project dynamics.' A flexible project still needs leadership - but leaders who act as facilitators rather than strict commanders. And the folks they lead need to be able to form working groups that have clear purposes and escalation paths to ensure corrective action can occur when needed (and not just on the little issues).

There are of course other factors involved in the failure of complex IT projects, but a large chunk of what tends to happen can be mapped to the issues raised in this article. In a future post, we will look at some of the warning signs associated with a project 'at risk."


Copyright 2013, Stephen Lahanas

Wednesday, May 22, 2013

How to Set Trends


IT is a very trendy industry. You might say we’re more or less slaves to fashion. The priorities which are set in company boardrooms and technology departments are quite driven by the set of 'must have' capability being hyped at any given time. We’ll forgo discussion as to whether this is a good thing or not for now and focus instead on the act of setting a trend rather than merely following one.

First of all, why would someone want to be a trendsetter in IT; well here are a few potential motivations:

  1. Because inventing something is fun, exciting or at the very least represents a break from the typical routine. 
  2. Because identifying a trend can result in benefits to the trendsetter. I’ll emphasize “can” here because living on the cutting edge often results in some blood loss. There are risks to being new, different or the first one out of the gate.
  3. Because creating your hype is and always has been good business. Not everyone can pull it off, but many of the most successful technology companies were built (at least initially) on hype.

Now, it may sound like I’m trivializing the topic a bit or perhaps implying that there is little substance behind most trends – but I’m not really. There can be a marriage behind trend-setting and true innovation if that’s the way you wish to pursue it. Obviously, some trends involve a much higher investment threshold than others. For example, inventing the next commercial quantum computing platform is going to require a lot of capital up front to pull off. However, there are lot’s opportunities hidden with the mass of existing technologies that remain unexploited as well. For most of us, the easiest path towards that exploitation is through invention of new practices.

Who defines the road ahead?
So what does that mean exactly? Practice, any practice, is the day to day approach that a professional takes to do their job. In Medicine, practice is the application of diagnostic and prognostic algorithms (rulesets and techniques) based upon understanding of a much wider body of knowledge. Individual practice in Medicine can influence the holistic practice of medicine across an entire society (or the world) through the publication of these new techniques and insights. Those new techniques if grouped together can also be viewed as trends. Some examples from Medicine of recent trends include:

  • Introduction of Electronic Healthcare Records 
  • Reductions in the amount and frequency of antibiotics being prescribed
  • Exploitation of genetic markers to predict, diagnose and treat a wide array of illnesses

Notice that in the above examples, two of the three trends listed were in fact technology-focused. This is an important consideration when thinking about becoming a trendsetter – nearly every industry now depends on IT to provide a significant amount of its innovation. This means that IT professionals have the opportunity to set trends not only for their own industry but every other one as well. This is exciting and extremely cool if you think about it. For the first time in perhaps forever – folks who weren't trained in specialized fields are helping to redefine those fields because of the technology being applied. It’s not uncommon for an IT consultant to move across industries from one project to the next – in each one learning the business enough to determine how best emerging technology might improve it.

Maybe the best way to describe how to become a trendsetter is through an example – so I’ll draw upon one from my own experience. In 2007, I was consulting for a firm that specialized in Business Intelligence and Data Warehousing. They asked me to determine if some of their solution offerings might be cobbled together into some sort of unified practice approach. I had worked with them before and seen their solution in action and had worked in that industry space quite a lot as well. So here’s what I did:

  1. I identified all of their strengths and characterized them in depth.
  2. I assessed the current practice approaches in industry (circa 2007) for BI and DW and identified a list of gaps.
  3. I mapped the identified strengths of the company against the industry gaps and as I suspected there were several key areas of alignment. In other words, there were things that the company was doing that wasn't commonplace in industry yet – but really needed. 
  4. I took that and expanded upon it; identifying a theme for a practice approach as well as a methodology and also identified areas where some technology gaps still existed (even within the new model). 
  5. This then led to a standard solutions architecture and detailed process approach based upon the theme and high level methodology. 
  6. I actualized this within the company buy presenting the findings and recommendations – many of which were focused on how to blend the new practice approach within the current business model.
  7. I then launched an outreach effort – through existing customers and social media. 

The practice I created for them was coined “Agile BI.” At the time, the reception was mixed. Very few in the industry were viewing Business Intelligence or Data Warehousing as something that could be considered Agile. This was before the explosion of Big Data and NoSQL solutions. The main premise I was promoting though is very close to what the industry is defining as Agile BI now:

  • That BI projects can be time-boxed into clearly defined increments (you don’t have to use Scrum per se). The key is that some capability is delivered within each increment. 
  • The approach was less product-focused and more capability focused. In other words, in this practice a wider variety of tools and capabilities were considered mainly because of the recognition that most environments have a wide variety of data needs / solutions.  
  • That BI and the underlying data structures needed to be engineered for performance from day 1.
  • That BI and DW were not IT only efforts but required very close collaboration at all stages with key business stakeholders. 
  • That new modeling or architecture techniques were needed (that extended beyond 3NF ERD) to help unify management of diverse data resources. 

I recall presenting this all in Chicago in 2007 at an industry conference and receiving a lot of blank stares. This of course is where the risk comes in; it takes years and some serious marketing to go from a kernel approach to widespread industry adoption. In this particular example – that actually has occurred - ( see The TDWI Agile BI Portal ). The company I introduced the practice to did not have the resources to wholeheartedly pursue the approach and stuck to their old business model and I moved on, but not before posting several presentations online and creating a blog that may have helped contribute to the general mindshare behind the industry premise. Granted – it was an obvious and logical progression that would have likely occurred anyway - but my contributions proved to be a worthwhile exercise none-the-less.  

One of the side benefits though of helping to shape industry-wide expectations of how technology practice ought to look like is that eventually you find yourself in projects that have adopted the trend you helped to set – which in itself is fairly satisfying – especially if you really did help to promote a better way of getting things done instead of just generating more hype.

Saturday, December 29, 2012

IT Predictions for 2013

Before we continue with our discussion regarding how to develop Conceptual Models, we're going to pause for a moment and look at the year ahead. In many ways, Enterprise IT seems to move a bit slower than the overall tech sector - that's often because the nature or mission of Enterprise Information Technology is the application of new technology rather than its invention (in most cases). Even still, the last two years have seen some fairly remarkably developments, including:

  • Industry-wide adoption of Cloud Computing models (both internally and externally)
  • Rapid movement towards the concept of Big Data (before that concept has been fully fleshed out).
  • Very rapid movement towards adoption of both social media and mobile applications (and integration of the traditional Enterprise IT with both).
Sometimes, the future is hard to read...

Two years is a quick turnaround and what it really represents is the turning of the ship in a new direction - as opposed to having reached the distant shores. While progress is being made in these new directions - they are in most places 'works in progress.' And of course they must be added to the backlog of works in progress that are already present - yesterday's hype-driven initiatives such as:
  • Cyber security
  • ITIL
  • SOA
  • EA
  • ERP (yes it's still out there and very much alive).
  • Business Intelligence
  • Semantic Technologies (the Semantic Web)   and so on...

Today's IT department is a busy place indeed - it is coping with more new technology expectations than ever before. Yet seldom do IT budgets reflect the widening scope of those expectations. Given this background we're going to make a handful of predictions for 2013...

  • Prediction 1 - Big Data will be redefined into a more actionable approach - this approach will become more architecture driven and less focused on roll-out of databases alone.


  • Prediction 2 - Big Data and Semantic Technology will begin to merge - this will occur through management of meta-data as well as through management of architecture-driven Big Data solutions.  
  • Prediction 3 - More project teams will be consolidated - much of this will happen onshore but some it will be near-shore and offshore. Why? Because more firms are beginning to understand that globally dispersed teams aren't that efficient - initial costs saved must be measured against overall lack of effectiveness in understanding client needs.
  • Prediction 4 - There will be some backlash against the frenzy of IT consolidation that's been occurring for the past several years (driven mainly by Google, Oracle and IBM). This may lead to more anti-trust actions.
  • Prediction 5 - There will be increasing pressure on Congress to examine H1b policy decisions. Most of the influence on these policies have been driven by a handful of the largest IT related companies - their arguments and contention that there is an IT skills shortage in the US will be called into question.
  • Prediction 6 - Mobile App development will become the hottest / most desired job title - at least near-term.
  • Prediction 7 - Massive Online Multi-player gaming environments will unveil new monetization strategies. Massively available online courses will do the same and move towards interchangeable content.

Of course, the most enjoyable part of our industry is how often we end up getting surprised - we're looking forward to that as well...


Copyright 2012  - Technovation Talks, Semantech Inc

Monday, December 3, 2012

The Semantic COP Motivation


The number one challenge associated with the entire domain of Information Technology is data overload. This problem is not merely a matter of increasing complexity, because even as other areas within IT are becoming more integrated or manageable, the ability to effectively exploit the increasing quantity of available data is decreasing. There are a number of reasons why this is the case; those reasons include:

·       Traditional data technologies, standards and related solution architectures were simply not designed to handle this much data. Data sources were built to exist as silos and most of them have not changed in principle for about 30 years.

·       The requirements for sharing data across organizations or across the globe is a recent expectation and changes all previous expectations regarding data management, integration and discovery.

·       Unstructured data is now becoming as important as structured data. This adds a nearly infinite curve to potential data growth.  

Several key capabilities are needed to help get control of data overload

For those organizations or individuals charged with performing sophisticated analytics-based tasks, the future will be challenging. While data continues to grow at nearly an exponential rate, the tools and technologies dedicated to making sense of ever-larger data sets is not keeping pace. This is not merely a military or intelligence-focused problem; it affects many commercial domains as well. The fact that this is such a common problem makes the prospect of a solution architecture designed to address this challenge more appealing and commercially viable. Such a solution could be presented both as specific commercial products and a best practice architectural approaches (not unlike SOA or Cloud Computing). While this solution may take advantage of Big Data technologies such as Hadoop it is not primarily a "Big Data" approach.

This challenge requires both a new architectural approach but also requires a new way of managing IT lifecycles to better exploit the emerging technologies which comprise that architecture. Semantic Technology is the key to facilitating a new type of analytics, one that has the power to harness any type of data source or architecture.  The overall solution approach that we will be proposing is what we're referring to as a Semantic Operating Picture. The primary solution elements which build upon the S-COP principles we presented yesterday are as follows:

  1. Development of a clear definition the Problem Space and related technical challenges.
  2. Conducting applied or directed research into specific areas:
    • Dynamic Tagging , Master Data & Metadata Management.
    • Current Semantic Standards; focus on RDF, OWL, RIF.
    • Semantic-based Visualization technologies; especially those using OWL, RDF.
    • Semantic Search and Inference Engine technologies.
    • Semantic Lifecycle Management capabilities
    • Natural Language Processing (algorithms, strategies etc.)
    • Translation of RDBMS or DW structures to Semantic Standards & DBs.
    • Semantic-driven reporting capabilities; integration w/ visualization.
  3. Identification of multi-modal datasets for use in the prototype solution architecture.

We will explore these in more detail in upcoming posts.


Copyright 2012  - Technovation Talks, Semantech Inc

Saturday, December 1, 2012

The Semantic Common Operating Picture

One of the most important achievements in analytics during the 2000's was the development of what can be described as Common Operating Picture architectures or COPS. These solutions combine data fusion technology along with advanced reporting functionality and role-based dashboards to provide an integrated view of various types of data. COPs are often used in the DoD or Homeland Security to support threat analysis or other Cyber Security missions. However, the current COP architectures are lacking in several key areas including:
  • Scalability (e.g. it's costly to make changes to the underlying integration)
  • Ability to manage Big Data
  • Accuracy across larger sets of disparate data (or domains)
For these reasons a new generation of COP solutions is needed. We're going to discuss one such idea today - the Semantic Common Operating Picture.

A standard COP Architecture
This approach represents a fairly ambitious, evolutionary step to the family of solutions already referred to as Common Operating Pictures or COPs. Sometimes these types of systems are referred to by their functional role, which in this case most often is described as Situation Awareness. Most COPs include a number of technologies within their solution architectures and thus represent a both a product (or set of products) and an integration activity.

The Semantic Common Operating Picture (S-COP) approach builds upon previous COP architectures but differs from them in several critical ways:


  1. It will take advantage of new standards in data management and message manipulation that weren't yet defined or matured when standard COPs first began appearing more than a decade ago. Many of these new standards are associated with the concept collectively referred to as the “Semantic Web.”
  2. It will allow for flexibility in how data is presented with more and richer opportunities for complex visualizations.
  3. It will support collaborative activities by design (and not as an afterthought). The potential associated with Collaborative Analytics was discussed in more depth in the previous section of this proposal.
  4. It will provide more opportunities for data discovery through a more flexible approach towards management of data sources. In other words, this solution will have access to more unstructured data and will give users the ability to add meaning to that unstructured data (and to structured data as well).   
  5. It will support the emerging field (and technical standards related to) Big Data.



The S-COP concept is based on a set of seven principles, listed in the mindmap below. 



These core principles supporting the S-COP concept are as follows:

·       The solution must support multiple levels of analysis – this implies the ability to manage analytical tasks at the personal level, the group level and the organizational level. This in essence is not too different from the traditional COP architecture. However, we can also view this principle in the context of data temporality. It is important that the solution be able to differentiate chronologically sensitive data. 
·       The solution must support complex collaboration – this implies that the solution must be able to facilitate not just simple messaging between participants in various domains, but must also facilitate complex collaborative analytics.
·       The solution must support interactive annotation – what this means in the context of this particular solicitation is the ability for end-users or subject matter experts to add meaning to existing data elements or sources through data tagging or adding other notes.
·       The solution must support multiple visualization capabilities – in the context of this project this means the ability to display both traditional dashboard-like graphics as well as complex concept visualizations. For example, it would be beneficial to be able to illustrate relationships between data without necessarily having to define specific queries or analytic representations.
·       The solution must support intelligent discovery – This means that the solution must have the capacity to learn from previous heuristic paths and results. Intelligent discovery must be able to harness existing subject matter expertise across partner domains as well as existing knowledge bases. It must also be able to exploit patterns.
·       This solution must support pattern management capabilities – This implies the ability to discern, record and manage patterns from what has been previously discovered to more quickly assess current situation awareness. Pattern management must be able to build patterns out of both structured and unstructured data.
·       The solution must support rules management capabilities – Having Rules Management allows for discovery and analysis to be linked in real-time to mitigation options. It also helps to automate more of the analytics processes thus allowing resources to focus their attention on only those areas which really need it.
These principles taken together represent a basic framework or requirements hierarchy which can be used both to assess possible technologies and to build a solutions architecture. We will examine the architecture of the S-COP in greater detail in future posts...




Copyright 2012  - Technovation Talks, Semantech Inc.

Friday, November 2, 2012

Understanding Data Warehouse Challenges

Last week we talked about some of the issues arising from the emerging field of Big Data, today we're going to explore a more mature data solution and point out some of the challenges that it's faced - and some of those challenges were instrumental in pushing the IT industry towards Big Data.

The Data Warehouse concept is built atop the notion that all data related to the enterprise can be captured and centrally or holistically managed. This is a powerful idea, yet there is more than one way to achieve that goal. The traditional view of the EDW attacked the problem from a very DBMS-centric perspective. This is primarily why EDW projects become so expensive, difficult and ultimately hard to adopt. The typical EDW approach attempted to gather all of the data related to the enterprise and place it into one massive repository structure. Whether this approach was attempted in chunks or as a “Big Bang” assault made little difference in the long the run as the byproducts of the practice were the same; those byproducts included:
  • A more bureaucratic management approach to the data layer in general.
  • An added degree of separation between the data owners and the data developers.
  • A certain level of inflexibility in regards to how data was updated, corrected or otherwise transformed.
  • An added degree of separation between database developers and data exploitation developers.
  • An added degree of separation between database developers and application developers.
  • An inability to quickly respond to major changes in the business.
  • Dependence upon a sub-set of industry experts and equipment that is more expensive than the industry norm.
  • A higher cost associated with scalability in general.
It is worth examining some of the core concepts associated with Data Warehousing a little bit closer to help understand why these outcomes tend to result from the traditional EDW approach.

DBMS Focus – At the time when EDWs became popular, other areas of data architecture were only just beginning to blossom. Today’s Business Intelligence platforms represent much more than mere reporting engines. Metadata management was only just beginning to be understood in the mid-1990’s and focus on Semantic technologies was virtually non-existent. The world according to DBMS in 1995 had a relational management system in the middle with ETL feeding into and reports coming out. This might be thought of as a three layer, stove-piped database systems view of the data architecture.

The Enterprise Single Instance – While consolidating like capabilities into marts or stores or some other ‘functional single instance’ approach has achieved quite a bit of success over the past two decades, attempting to manage all data in one structure has proven much more difficult. This is why the notion of Massively Parallel Processing (MPP) was needed to make it viable back in the 1990s. MPP in the context of proprietary hardware was expensive though and perhaps failed to recognize the power of networked processors on inexpensive hardware (i.e the Google scalability model). The other key consideration here was the added steps that were needed in order to make such a system perform within reasonable parameters. So, the single instance enterprise faced and still faces major hurdles in terms of costs, manageability and performance. 


Data is no longer confined within the context of single systems
EDW Fallacies
If we were to directly challenge the core EDW assumptions and illustrate the fallacies associated with the philosophy, our list would resemble the following:
  • The Business will remain static over a relatively long period of time.
  • The Enterprise will remain static over a relatively long period of time.
  • That source data and data exploitation should not be managed synergistically, in other words that Decision Support or Business Intelligence solutions built on top of EDW source data should be viewed as separate, albeit related efforts.
  • That the data layer and the application layer can or should be viewed or designed separately.
  • That computer Hardware would not catch up to the processing load – i.e. that the data layer would always require specialized Massively Parallel Processing (MPP) in order to manage very large quantities of data. Furthermore, this assumption also implied the data would remain in a single instance data source. So instead of parallel processors deployed in specialized equipment, Big Data now uses the cheapest processes / equipment possible in a commodity approach with data spread out in sets of distributed file systems. In fact this has been take even further as this week the US Government announced the completion of the world's most powerful supercomputer. It achieved all of its latest gains by using commodity hardware (in this case, GPUs, game video processors widely available on the market).
  • That network architecture, data architecture, application / SOA architecture and enterprise architecture are separate.
  • That the Internet (Cloud) would not represent a viable mechanism for connecting to distributed data sources.
  • That unstructured data was not as valid as structured data (mainly because no mechanism existed to incorporate into the traditional database management approaches).
  • That most major transformations need to occur before data is placed into the primary storage / management entity (i.e. DBMS, warehouse).
  • That there is a single version of the truth, period. This is perhaps the biggest fallacy behind all data warehouse, MDM and Governance solutions. Data can be managed, but it is dynamic and all always will be. Viewing data as incontrovertible, orthodox truth immediately eliminates much of the value that data otherwise provides. Situations change, and every stakeholder views the whole from their unique perspectives. Yet, there can still be order in a relativistic environment (much as there is in the real world).  This doesn't mean data cannot be standardized or managed - it merely takes into consideration the inevitable evolution that will occur.


Copyright 2012  - Technovation Talks, Semantech Inc.