The renowned American market research firm Forrester Research propounded the idea of big data fabric in 2016. According to Forrester, big data fabric “accelerates insights by automating ingestion, curation, discovery, preparation, and integration from data silos.” While it is already a buzzword among enterprise architects and data enthusiasts, what does the advent of data fabric mean for you? How can it help your business? Is it merely an extension of data warehouses and data lakes? Or is it a complete paradigm shift? In this post, we will answer some of these questions and fast-track your journey towards implementing this powerful, omni-potent framework in the context of your enterprise.
Need for Data Fabric
With the rapid and exponential growth in data variety, velocity and variability, data preparation and management has become an excruciatingly painstaking process. The vast majority of enterprise data today is unstructured, residing outside the organization, thereby shifting the centre of gravity of enterprise data to an external locus. This shift has huge implications. Firstly, external data is rarely clean. It comes in the form of a data ore that is contaminated with substantial noise, making it sparse and non-uniform. It is also largely unstructured. Cleaning up this data through multiple stages of pre-processing is a battle of its own.
Secondly, this external data needs to be married to the organization’s internal data. Here, the challenge is to seamlessly unify external and internal data that are available in diverse and incompatible formats. Merging of disparate data forms into a single, machine-readable and human-retrievable format is a necessary evil for any organization trying to derive value from their data. Unfortunately, doing so has enormous cost and time implications for businesses.
Even if you can solve the above puzzle of siloed, incoherent data by investing extensively in some of the most expensive resources, you will still be heavily dependent on inefficient legacy systems underlying your IT infrastructure. This means that your ability scale up and down as per your organization’s needs will be severely compromised. Mobility of data across data centers, edge devices, and cloud instances will come at the cost of tampered security, while costs and delays will spin out of control at the same time.
Finally, data needs to be enriched with knowledge layers to become contextual and meaningful with respect to the underlying business domain and must be hydrated with fresh inflow of new data and insights in real time to stay relevant, useful and meaningful. Without this, data and the complex relationships encapsulating it cannot be made accessible to decision makers in a secured manner that preserves its lineage, integrity, and resourcefulness.
This is where data fabric comes in.
What is Data Fabric?
Contrary to popular belief, data fabric is not a mere spatial extension of data warehouses and data lakes. It is a transformative approach in envisioning enterprise data that revolves around the need for a single version of the truth - or at the very least, for only a few compatible versions of the truth. Imagine a data management framework that allows you to securely store and access data with flexible granular controls, easy mobility across edge computers, coupled with seamless version control across the entire repository. The smartest form in which data can be cast in such a wide and versatile structure is the graph data model or the smart enterprise graph. This smart enterprise graph or data fabric can then be modeled and purpose-built to turbo-charge pattern detection, free form queries and powerful AI algorithms.
This smart enterprise graph is then continuously updated with a smart grid of new data, insights and domain relationships. It can be compressed and decompressed to any degree of granularity and can even be interspersed with IoT, Blockchain and much more. This is particularly significant as Gartner predicts that 90 percent of the organizations will adopt hybrid cloud infrastructure management capabilities by the year 2020.
What Can You Do with Data Fabric?
By adopting the data fabric approach for your enterprise architecture, you can navigate fluidly across disparate data sources and infrastructure types. With a single, consolidated framework to manage, enhance and connect data powered by mobility across multiple isolated decision centers, you can leverage infrastructure solutions that align with your business needs without worrying about compatibility, integrity or security. This cohesive, smart analytics grid cutting across multiple cloud and on-premise environments will enable you to:
- Access and unify disparate, siloed data sources in real time
- Implement centralized data management and processing
- Protect enterprise data with banking grade security and recovery protocols
- Bring about consistency, integration and versatility in hybrid cloud environments
- Optimize data and IT investments, reduce sunk costs caused by technology and data obsolescence
MECBot’s Smart Data Fabric for Augmented Data Management
Simply put, MECBot automates the entire data unification process envisaging all forms of data at scale, and delivers unprecedented business results. To accomplish this, MECBot first structures the unstructured data contextually using domain specific business ontologies and marries it with structured transactional data in near real time. This creates a complex Data Graph for an enterprise (Smart Enterprise Graph).
This Smart Enterprise Graph is accessible through “Free Form Vertical Search”, APIs, SparkSQL and Graph QL. Plus, it is always hydrated through schedules and can be transformed into any shape required by the downward analytics or data application layers. MECBot also comes bundled with cool features like teleporting to wherever you want, banking-grade security, version controlled data through pachyderm and also OOB ML/DL/AI/Statistical/Graph algorithms that run as functions (FaaS) at Scale using Kubernetes and Dockers.
What's more - you can say goodbye to tiresome setups. It takes just a single click installation to create MECBot clusters in Public Cloud (Amazon, Azure, GCP etc.) or on On-Premise and comes bundled powerful monitoring tools to monitor the entire cluster. MECBot's unique value propositions include:
- Simplicity: Sets up the application with just a few clicks. Gives results from day 1.
- Speed: Matches the pace of decision-making with the speed of the original data generation.
- Scale: Performs elastic scaling based on business needs with dynamic clustering.
- Security: Offers banking grade security and role-specific access to the platform.
MECBot saves more than 80% of Pre-processing Time & Cost, and delivers highly actionable insights to boost your ROI manifold. It is the only platform that puts your business first, not data.
Its unique, built-in capabilities drastically reduce the burden on IT infrastructure and empower your decisions with powerful business intelligence in real-time through augmented data management.
Read our blog on Augmented Data Management.
How It Works
- Data Ingestion: MECBot first ingests all the data pertaining to the enterprise both from external and internal sources. These data are available in diverse forms, shapes and sizes. Unlike traditional analytics, MECBot is equipped to ingest data in any form, be it Structured Data (e.g. Databases), Semi-Structured Data (e.g. Excel files, .csv files, etc.) or Unstructured Data (e.g. Blogs, Social Media, Reviews, Chat Data with Customer Service Agents, Documents and PDFs, etc.)
- Data Preparation: Once ingested, MECBot massages, folds, pre-processes and cleans the data. This is a particularly critical step that is automated by MECBot which otherwise would require expensive man-hours of data scientists and cause significant delay. In fact, if this pivotal step is skipped, further analysis will lead to faulty results and poor business decisions, as the raw data itself is polluted with noise.
- Data Harmonization: MECBot’s flagship capability is to unify all the pre-processed data into a single form, shape and size. The extraction of ‘Facts’ takes place on structured data, semi-structured data as well as unstructured data. In the former, the Fact that emanates is a tabular representation of data relationships, while in the latter, MECBot’s text analytics tool annotates the unstructured text data into an assortment of industry-specific keywords that are connected to each other.
- Data Enrichment: Data without an appropriate frame of reference is meaningless. However, today, business users and citizen data scientists struggle to add the appropriate context to their unified data to glean the correct meaning and relationship from it. MECBot’s Smart Enterprise Graph is uniquely equipped with a scalable Enhancement Engine and a scalable Knowledge Base that take care of this at three distinct levels: Universal Knowledge Base (Wikipedia), Domain Knowledge Base and Tribal Knowledge Base.
- Context Lattice: Simply put, a context lattice takes user inputs on the data relationships and his preferences to extract logical datasets that unravel these connections. These logical datasets or the context lattices can be accessed through MECBot APIs. All the data are hydrated in real time to enable real-time updated view for the business users.
- Flattened View: A context may span across multiple entities. It can also create relationship boundaries between entities for a given business context. To enable downstream analytics such as data visualization using tools like Tableau, R, etc., this context lattice is transformed into a more functional flattened view, or a tabular representation of the contextualized data.
- Free-Form Search: The data is now ready to be analyzed, and MECBot is now available to the user to slice and dice the data. You can say goodbye to multi-op queries and SQL Queries with our free-form search.
- Explore data with descriptive analytics tools, like range, standard deviation, central tendency, etc.
- Fully functional query in English language from multiple databases.
- Visualize the data through third party tools like Tableau, Qlik etc., or use MECBot’s own visualization tools - Analytics - the Brain of MECBot, Powered by AI: You can now perform advanced AI functionalities like discovery of patterns and insights using Machine Learning, Deep Learning and NLP, use pre-built AI models with a single click, such as Customer Loyalty, Segmentation, Clustering, Predictive Analysis, and so on.
You can also perform highly specific queries on your data that can further augment your insight pipeline and user dashboard. Enhanced free-flow search can also happen now, like "Customer Loyalty in Bangalore," where loyalty determination is enabled via Customer Loyalty model. - Smart Insight Grid: The insights generated themselves become a continuous grid of discoveries that keep refreshing in near-real time and keep streaming to you a dynamic picture of the data problem at hand. This smart, continuous insight grid runs in the background and feeds all your business decisions at scale, reducing your time-to-market drastically and creating your very own reliable digital data team that ensures that all your decisions are data-driven. In this way, MECBot brings to you unprecedented business results, redefining business intelligence as we know it.
Interested to know more about how MECBot can boost your RoI manifold with Smart Data Fabric for Augmented Data Management? Visit formcept.com/products/mecbot/. To know about the state-of-the-art technologies we use, check out our platform architecture here: https://formcept.com/products/mecbot/platform/
Wish to take a deep dive into what MECBot can do for your business? Request a demo here: https://formcept.com/contact/
Refrences:
-
https://www.forrester.com/report/The+Forrester+Wave+Big+Data+Fabric+Q2+2018/-/E-RES141570
-
https://tdwi.org/articles/2018/06/20/ta-all-data-fabrics-for-big-data.aspx
-
https://www.bmc.com/blogs/data-fabric