MECSense | MECBot’s Patented Datafolding Technique Helps Businesses Auto-Detect Patterns from High-Volume, High-Velocity Data in Real-Time

By 2020, the world is projected to sit on top of ~40 trillion gigabytes of data. When organizations attempt to mine such a large volume of data moving at high velocity, they face four key challenges:

Effective & efficient retrieval of the right data at the right time
Cleaning noise and doubtful data to extract high-veracity data (or accurate data)
Ensuring that this piece of data is constantly updated while preserving its relationships with other data
Detecting patterns in the data automatically and extracting actionable intelligence in the form of insights and alerts in real-time in a scalable manner

MECSense | Making Sense of Data Through Automatic Pattern Recognition in Real-Time

Unlike traditional data modeling techniques, auto-detection and recognition of patterns do not rely on the decision-maker, data-scientists or any human entity to model the answers to a particular question pertaining to a domain. It is not limited by underlying datasets, schema or algorithms. This means that instead of deploying specific algorithms that are trained to detect only a certain kind of patterns (trained or supervised machine learning), the data is automatically and dynamically analyzed by folding the related or similar data-points into context lattices that identify patterns intelligently to generate actionable insights in real-time (unsupervised machine learning coupled with cutting-edge artificial intelligence).

FORMCEPT deploys its patented Datafolding technique, MECSense, through its flagship product MECBot which is the #1 Augmented Data Management Platform for Real-Time Analytics at Scale. A datafold tells you the most important data points in a dataset that may be relevant for your business and which you might miss or oversee even after prolonged self-exploration of the data. MECBot folds the data across various data points or nodes like origami to automatically reveal the actual patterns hidden underneath. These data points might span across multiple disparate datasets having interlinkages across multiple other related data points. Hence, in order to generate accurate and comprehensive datafolds, MECBot takes a business-first approach (as opposed to data-first approach by traditional analytics).

This is achieved by using the Entity Domain Model approach, which first aligns all the data points according to the domain in context. This way, MECBot operates without any dependency on the underlying databases or the structure of the data. It comes bundled with a self-service, intuitive interface, and takes care of all your data management and analytics requirements in a centralized manner, including scalable deployment. There are two ways to fold the data, which happen simultaneously in MECBot. These are ‘classification of patterns’ and ‘clustering of patterns.’ Classification refers to contextualizing the pattern based on an abstraction that points to a particular domain. Clustering refers to creating folds or partitions within the data on-the-go that directly affect a decision variable. It is used to identify how patterns in the data relate to one or more decision variables in a given time.

Use Case and Platform Demo: MECSense for Retail Industry

MECBot’s datafolding function democratizes the insight generation process by making repeated self-exploration redundant through smart data discovery. Users can specify one or more data points for MECBot to generate datafolds along those points and reveal patterns and explanations across them automatically in real-time. This means that the user doesn't have to limit the insight generation process to a set of predefined questions or hypotheses.

Let us take the example of a retail use-case. The domain ‘retail’ can be defined by means of multiple entities such as items, products, locations, brands, markets and so on. Each entity can then be described through attributes. These relationships between retail entities and their attributes can be captured in MECBot through an Entity Domain Model (EDM).

Step 1: Applying Filters to Set the Context

In order to generate insights and patterns automatically on MECBot through datafolding, first, the user would have to apply a set of filters to specify the boundaries of the dataset.

In the screenshot below, for example, 7 filters have been applied to the product entity (items_product_id), which are butter, flour, salt, sugar, rice, milk, and pulse.

Similarly, filters can be applied to other entities like markets, brands, locations, etc.

Step 2: Selecting Columns or Data Points

To create datafolds using MECSense, we first need to specify the columns or data-points across which we want to generate patterns and insights. After applying the filters, let’s click on ‘Create Folds’ on the left corner of the screen, as shown in the screenshot below.

Now, let’s select the columns for datafolding. As shown in the screenshot below, the first column selected is brand (items_brand_name).

This way, we select more columns for data folding like name of item (item_name), name of product (items_product_name), location (location_name) and market (market_name).

Also, select the statistics which we want to compare among the selected data-points. Here, let's select the average of Amount, User Age and Rating.

Step 3: Naming the Datafold

With MECSense, we can create multiple datafolds by selecting unlimited combination of columns. Naming the datafolds can help organize our insight generation process for quick discovery.

Let’s name our current datafold as product_folds, for example.

Step 4: Insight Generation Using Datafolds

MECBot now detects patterns and generates insights automatically by applying datafolding algorithm across the selected columns. Further, the user can generate need-based insights on specific entities. For example, the screenshot below shows how MECBot folds the available datasets across the entity market (i.e. the column ‘market_name’) to show patterns across transaction amount, customer rating and age of customers in markets like Spencer’s, More, Hypercity, D Mart, Big Basket and Organic World.

We can also combine multiple columns or data points in MECBot to generate insights through datafolding. For example, the screenshot below shows the results of datafolding across market name, location name and brand name. For example, here we can see that the highest amount of transaction has been in Spencer’s outlets in Ahmedabad for ITC brand.

Further, the user can also dive deeper into the generated datafold by querying for a specific entity. For example, the screenshot below shows the insights generated by MECBot on querying for the location ‘Ahmedabad’ in the above datafold. We can see that even though Spencer’s emerged as the retail outlet with maximum transaction in Ahmedabad in the previous datafold, the highest customer rating among all the markets in Ahmedabad has been obtained at Big Basket for the brand ITC. The is shown in the datafold screenshot below.

This way, we can carry out multiple layers of pattern detection and smart data discovery in MECBot through datafolding, and automatically generate valuable, actionable intelligence that can help us make informed decisions for maximum impact.

Use Case: MECSense for Medical & Pharmaceutical Industry

Let’s consider the case of a pharmaceutical manufacturing company which has launched a new analgesic product ‘PainZy’ in the U.S. market with the key value proposition of instant pain relief. By creating an entity-domain-model on MECBot, the company can upload the data on this product across markets, geographies, customer segments, distribution channels, and so on.

The data can be:

structured (e.g. sales data of PainZy, marketing spend across channels, etc.)
semi-structured (e.g. customer ratings, reviews, etc. ), or
unstructured (e.g. articles on competitors, reports, etc.)

MECBot’s in-built datafolding framework correlates and interlinks diverse data points to establish patterns and generate smart, actionable insights, like:

75% of customers tried out PainZy in California and Delaware where Competitor A is the market leader, while
12% of customers tried out PainZy in Florida and Pennsylvania where Competitor B is the market leader.
40% of those who tried PainZy suffered from side-effects like nausea and restlessness, and further, the majority of side-effects were reported by customers who drank coffee immediately before or after consuming PainZy.

Based on these insights, the pharmaceutical company can create more personalized promotional campaigns to differentiate its value propositions better from Competitor B and incentivise distribution channels in markets where Competitor B is in the dominant position. The company can also improve its product to mitigate the adverse effect when combined with beverages like coffee, and in the meantime also modify its dosage instructions to indicate that PainZy should not be taken immediately before or after consuming coffee.

Datafolding and Pre-processing in MECSense: How it Works

The process of Datafolding is carried out in two stages. First, the data is pre-processed and stored based on its type (such as entity dataset, event dataset, domain-specific association dataset) at the time of storage.

Then, after applying the business domain context & relevant knowledge bases and flattening of the data, MECBot folds the data across domain-specific data points automatically to enable automatic pattern recognition. We have already discussed the second aspect (i.e. auto-detection of patterns) in the paragraphs above. We will now take a quick look at pre-processing in datafolding.

During pre-processing, the system extracts all the relevant information such as entities, events (time-stamped data) and domain-specific associations from the content and stores them separately - each in a different data store. For instance, entities are extracted from the received content and then stored in an entity store, events are extracted from the received content and the extracted events are stored in an event store (designed to address end-user queries based on time ranges).

The system then creates a data-lattice using the identified entities, events (time-stamped data) and domain-specific associations. It stores the data lattices as views and then stores the views back into the view store, wherein the view store is optimized for querying these views. The system stores the created data lattices as a view so that the system queries the stored views based on the requirement of the end-user. The data lattices so created and stored are further folded to create higher levels of data lattices.

Key Advantages and Benefits of MECSense:

Auto-detects patterns without the need for human intervention, intricate modelling or force-fitting underlying schema and data structures
Deploys exploratory analysis and advanced analytics modules that are built on top of a smart enterprise graph that captures your business domain in the most comprehensive manner
Allows you to create flattened data view for the chosen entities without writing any complex SQL joins

About MECBot:

MECBot is a revolutionary big data analytics product powered by AI which automates the entire data unification process envisaging all forms of data at scale and delivers unprecedented business results. To accomplish this, MECBot first structures the unstructured data contextually using domain-specific business ontologies and marries it with structured transactional data in near real-time. This creates a comprehensive Smart Data Graph for an enterprise (Smart Enterprise Graph). This Smart Enterprise Graph is accessible through “Free Form Vertical Search”, APIs, SparkSQL and Graph QL. What’s more – you can say goodbye to tiresome setups. It takes just a single click installation to create MECBot clusters in Public Cloud (Amazon, Azure, GCP etc.) or On-Premise and comes bundled powerful monitoring tools to monitor the entire cluster. MECBot saves more than 80% of pre-processing time & cost and delivers highly actionable insights to boost your ROI manifold. It is the only platform that puts your business first, not data. Its unique, built-in capabilities drastically reduce the burden on IT infrastructure and empower your decisions with powerful business intelligence in real-time through augmented data management.

Interested to know more about how MECBot can boost your RoI manifold with Augmented Data Management? Visit formcept.com/products/mecbot/. Wish to take a deep dive into what MECBot can do for your business? Request a demo here: https://formcept.com/contact/