The Semantic Common Operating Picture

One of the most important achievements in analytics during the 2000's was the development of what can be described as Common Operating Picture architectures or COPS. These solutions combine data fusion technology along with advanced reporting functionality and role-based dashboards to provide an integrated view of various types of data. COPs are often used in the DoD or Homeland Security to support threat analysis or other Cyber Security missions. However, the current COP architectures are lacking in several key areas including:

• Scalability (e.g. it's costly to make changes to the underlying integration)
• Ability to manage Big Data
• Accuracy across larger sets of disparate data (or domains)

For these reasons a new generation of COP solutions is needed. We're going to discuss one such idea today - the Semantic Common Operating Picture.

A standard COP Architecture

This approach represents a fairly ambitious, evolutionary step to the family of solutions already referred to as Common Operating Pictures or COPs. Sometimes these types of systems are referred to by their functional role, which in this case most often is described as Situation Awareness. Most COPs include a number of technologies within their solution architectures and thus represent a both a product (or set of products) and an integration activity.

The Semantic Common Operating Picture (S-COP) approach builds upon previous COP architectures but differs from them in several critical ways:

1. It will take advantage of new standards in data management and message manipulation that weren't yet defined or matured when standard COPs first began appearing more than a decade ago. Many of these new standards are associated with the concept collectively referred to as the “Semantic Web.”
2. It will allow for flexibility in how data is presented with more and richer opportunities for complex visualizations.
3. It will support collaborative activities by design (and not as an afterthought). The potential associated with Collaborative Analytics was discussed in more depth in the previous section of this proposal.
4. It will provide more opportunities for data discovery through a more flexible approach towards management of data sources. In other words, this solution will have access to more unstructured data and will give users the ability to add meaning to that unstructured data (and to structured data as well).   
5. It will support the emerging field (and technical standards related to) Big Data.

The S-COP concept is based on a set of seven principles, listed in the mindmap below. 

These core principles supporting the S-COP concept are as follows:

·       The solution must support multiple levels of analysis – this implies the ability to manage analytical tasks at the personal level, the group level and the organizational level. This in essence is not too different from the traditional COP architecture. However, we can also view this principle in the context of data temporality. It is important that the solution be able to differentiate chronologically sensitive data. 

·       The solution must support complex collaboration – this implies that the solution must be able to facilitate not just simple messaging between participants in various domains, but must also facilitate complex collaborative analytics.

·       The solution must support interactive annotation – what this means in the context of this particular solicitation is the ability for end-users or subject matter experts to add meaning to existing data elements or sources through data tagging or adding other notes.

·       The solution must support multiple visualization capabilities – in the context of this project this means the ability to display both traditional dashboard-like graphics as well as complex concept visualizations. For example, it would be beneficial to be able to illustrate relationships between data without necessarily having to define specific queries or analytic representations.

·       The solution must support intelligent discovery – This means that the solution must have the capacity to learn from previous heuristic paths and results. Intelligent discovery must be able to harness existing subject matter expertise across partner domains as well as existing knowledge bases. It must also be able to exploit patterns.

·       This solution must support pattern management capabilities – This implies the ability to discern, record and manage patterns from what has been previously discovered to more quickly assess current situation awareness. Pattern management must be able to build patterns out of both structured and unstructured data.

·       The solution must support rules management capabilities – Having Rules Management allows for discovery and analysis to be linked in real-time to mitigation options. It also helps to automate more of the analytics processes thus allowing resources to focus their attention on only those areas which really need it.

These principles taken together represent a basic framework or requirements hierarchy which can be used both to assess possible technologies and to build a solutions architecture. We will examine the architecture of the S-COP in greater detail in future posts...

Copyright 2012  - Technovation Talks, Semantech Inc.

Read More

The Trouble with "Big Data"

How can there be trouble with one of the two biggest trends in IT you ask? Well, perhaps from a hype and marketing perspective there isn't any trouble yet. But from an expectations perspective, the trouble began nearly two years ago and has only gotten worse. And it starts in a name, it sounds simple, but is it?

Can you define "Big Data" and if so would your define match an industry standard expectation?

What is Big Data, anyway? Well, this is where the trouble begins - it means something different to a fairly diverse set of interests. For some, Big Data implies use of a parallel processing paradigm (which BTW has been used for more than a decade in Data Warehousing as well), use of commodity hardware and a clever algorithm created by Google about a decade ago to help index the web. Much of this is now combined with the use of "Hadoop," although use of Hadoop doesn't always imply that companies will follow the same hardware path as some of the giants who pioneered the paradigm. In fact, more of than not the real market for commodity hardware is moving to the Cloud. But wait, aren't we talking about Big Data? What's the relationship between Big Data and the other biggest trend in IT today, Cloud Computing? Are they really two separate trends or variations of the same trend? The answer to that question is - who knows.

There are some other problems with Big Data; let's review them:

1. It seems to encompass a wide range of emerging technologies, such as storage, parallel processing, cloud technology, high performance discovery, new DBMS paradigms and more. 
2. The Use Cases for Big Data tend to blur into the same set of Use Cases for most enterprise data related functions. This wasn't always the case - the original Google exploitation its technology was fairly narrow and unique to its business model / mission. It sill isn't entirely clear how smaller enterprises will harness the newer Big Data capabilities - that clarity is vital - especially in regards how to integrate within the existing ecosystem.
3. There is no universally accepted definition for what it represents, but just as important, there is no recommended solution approach or set of approaches or even a recommended solution methodology. The largest IT trade group dedicated to Data Management, DAMA, has barely scratched the surface as to how integrate Big Data within the larger set of Data Management activities. Or should we assume that Big Data will somehow eventually swallow all of rest of what we were viewing as Data Management?

Let's step back in time for moment. Back in early 1999, I attended a technology conference in Washington D.C. that was convened to assess emerging technology trends for the next decade and beyond. One of the most interesting discussions that occurred during their main panel revolved around a question on how much bandwidth or data would be utilized in coming years. Recall, that in 1999, having a Terabyte of memory in a DBMS was  big deal and few if anyone had DSL like speeds for Internet access. The majority of the panel did not see any explosive growth happening in the foreseeable future. I disagreed - I countered that the demand had already been pent up and that a torrent of Digital content and communication would explode as soon as the hardware prices and bandwidth allowed. It's this exponential growth in data that the proponents of Big Data expound upon a lot these days (supposedly 2/3 thirds of all data ever created was generated in the last two years).

Well, guess what - that exponential data growth was merely a drop in the bucket to what's coming. And if that is truly the case, then we have to ask ourselves what this really means. Sure, we needed more affordable hardware and more affordable software to handle volume; we needed better algorithms and architecture to handle performance. The thing is though, we still haven't defined what this all means in relation to how we manage the enterprise. We've still got and we continue to support all sorts of legacy architectures and approaches - and now we've been handed a whole new set of challenges. But those challenges aren't just focused on bigger, cheaper, faster - we also have to deal with smarter, integrated and targeted. And we also have to become a bit more visionary when it comes to imaging what we can and should do with emerging worlds of data and that may take us right back to another set of technologies that have been emerging over the past decade right alongside Big Data - Semantic Technology. 
 
So, let's ask ourselves again:

1. Is Big Data about handling larger volumes of data faster?
2. Is Big Data about making Data Management more efficient / less expensive?
3. Is Big Data about harnessing the Cloud and Storage?
4. Is Big Data about expanding Data Discovery to cover ever-increasing sets of data?
5. Is it all of the above and / or something else?

Perhaps it's time for so more or better definition. Without better definition it will likely be to understand what the ROI is that you're shooting for or whether your organization is actually achieving it.

Copyright 2012, Semantech Inc. All Rights Reserved

Read More

Cyber Security & Threat Management

Threat Management is still a relatively new concept; there is no industry standard definition for it. In fact, the few people who are talking about it right now tend to view it from at least two very different perspectives – one a product focused approach to unifying perimeter security tools and two, a practice-focused management paradigm. As it evolves, Threat Management will eventually encompass both of those perspectives and will likely become perhaps the single most important element within any given Cyber Security solution.

The reason why it will become so critical is that Threat Management allows us for the first time to build upon a complex conceptual framework with a variety of analytical tools which will automate an ever-growing percentage of Cyber Security tasks. Without this framework it would remain difficult or nearly impossible to manage Cyber Security in a proactive and coordinated manner. For the purposes of this discussion let’s define Threat Management as:

“The conceptual and technical framework dedicated to discovering, defining and managing threats to operational security and mission assurance. Threat Management is software & hardware agnostic and can apply as an integrated IT practice in any functional domain. The goal of Threat Management is not merely to ensure that immediate (local domain) threats are mitigated but that threats are also managed in the context of communities of interdependent or inter-related entities. Threat Management depends upon top-down, bottom-up and lateral participation or guidance to build knowledge frameworks which can then be used to define security policy and solution mitigation.”

So, what is a “Threat” given this construct? A Threat is “any event, vulnerability or behavior (or combination thereof) that either poses a danger to the operational mission or if combined with other events, vulnerabilities or behavior could constitute a threat to the operational mission.”

The first step towards identifying threats is to define what threats actually represent

In that last sentence we begin to see the systems implications of what we’re talking about. The goals here are two-fold; one – block a threat before it is manifested or two – stop a threat in motion that wasn’t blocked in time to preserve operational capability. The other key consideration here is that we’re viewing this practice as evolutionary – it learns as it goes and learns from the community which uses it.

Threat Management and Semantic Technology
Much of what we’re describing with Threat Management already occurs in some fashion; however that is not consistent from one enterprise to another and in fact much of it is handled using manual processes with little ability to correlate or manage various aspects of the problem in a unified approach. To unify Threat Management we need a mechanism which allows us to characterize all aspects of Threats and to correlate that information from information collected from the full spectrum of security related software or hardware appliances. 

Threat Management as we’re describing it here is wholly dependent on a Semantic Knowledge layer and data exchange architecture. This allows us to:

• Provide non-proprietary data exchange approaches (for security-related data capture and analysis).
• Characterize complex or aggregated data “patterns” in utilizing ontologies or RDF-based databases or related tools.
• Provide a knowledge sharing framework for the community of defenders and security experts who analyze existing or predict future threats.
• Build policies based upon Threat Activity and Threat Prediction – policies that can also be captured, manifested and distributed using Semantic technology.
• Drive dynamic reconfiguration of H/W and S/W infrastructure in response to policy definition and distribution.

While there are Security vendors that have made incredible progress in being to integrate some of these capabilities in the context of their proprietary tools, this approach ultimately will fail without the Semantic layer for one simple reason – the entire world is never going to standardize on one security tool. However, the Semantic Layer for Threat Management can extend to encompass any infrastructure or combination of security tools.

Copyright 2012, Semantech Inc. All Rights Reserved

Read More