Article reference matching, enabler of a manufacturer data strategy

By -


In Manufacturing and Retail, the products are the major assets whether they are designed, produced, bought or sold.

In the typical product lifecycle, product related data are produced all along the lifecycle. Data can be design documents, manufacturing data, quality data, product sheets, ERP supplier data or customer data. The usage and end of life also produce maintenance.


All these data can be structured or unstructured contents. The digital Thread aims at providing a unified view of this lifecycle in order to improve efficiency, quality and/or reduce risks.

An Operational Datahub is a key component to enable this vision as it provides a convergent (Operational and Analytical), contextual (leveraging semantics) and data-centric solution to consolidate and consume all these data in real-time. All this being highly secured and cost-effective.

Datahub strategy provides quick wins use cases which can be :

  • Find nearest available stock whatever the unit is storing it
  • Find on which projects a reference and its alternatives are used
  • Optimise purchase strategy by consolidating orders and/order suppliers based on different references for the same part
  • Consolidate quality measures whatever the supplier or manufacturing site is
  • Etc.


The datahub is a masterpiece but it's not enough. It requires some smart mastering to deliver the value.

The challenge

If you consider the data produced along the lifecycle, it's usually not related to a unique unified reference. A part will indeed have multiple identities whether it's supplier reference, reference from manufacturer or even the reference under which it's sold to the customer. All these references are related to the same physical part but are all different, and all data from PLM, ERP or e-commerce platforms are attached to these various references.



So the challenge, if we want to leverage data collected in the data hub, is to match these references and perform transversal queries which reconsolidate all the data.

The solution

We will now explain how MarkLogic can leverage it's core capabilities to deliver the expect value.
The MarkLogic data hub foundations can indeed be natively enriched to provide a reference matching capability that will enrich any interaction application have with the datahub.




So the approach is to extract relations at system level to then merge everything to perform global matching

System level reference matching

At system level, it's usually quite easy to find related references and business owners are able to extract the rules that defines relations between references. It can be the customer references in order items, supplier references in part catalogs instances of designs in PLM system. With such knowledge, it's possible to generate the following graph of relations at system level:

Ingestion flow in MarkLogic

In order to implement this system level references in MarkLogic, we can leverage the Datahub Framework. First the source data (from ERP, PLM, etc) are loaded into the Staging area. From there, we can apply a logic based on business owners rules that will extract the references and their relations (customer ref., supplier refs, etc.) as golden records. It creates master version of a reference and its related references from a particular system. MarkLogic TDE (Template Driven Extract) can then create triples from these golden record in order to lift a graph of relations. At this stage, the graph is still related to a particular system.



From these relations you can then deduct that related references are actually alternatives. Depending on the process context, one reference can be used instead of the other.
In MarkLogic you could for example infer based on the main ontology that references have alternative relations between them (as soon as they have one of the typed relation)

Global matching

As soon as each individual system projects its known relations, if you combine the relations coming from all systems all along the life cycle, you get potentially a huge graph consolidating all this knowledge.


Now with a single simple SPARQL query that looks like ?sourceRef <alternative>* ?AnyAlternative we can get back all the references with are directly and indirectly linked to the initial reference.
It means that in a single query you can start from a deprecated reference, find the replacement references and for example all orders linked to any of these references.


Opportunities

The reference matching logic can then be attached to any functional query performed in the datahub.
The datahub being a operational system, it can feed in real time business processes to provide more part related insights:
  • Find nearest available stock whatever the unit is storing it
  • Find on which projects a reference and alternatives are used
  • Optimise purchase strategy by consolidating orders and/order suppliers based on different references for the same part
  • Consolidate quality measures whatever the supplier or manufacturing site is
  • Etc.

Moreover the same logic can apply to match supplier reference between systems and at scale.

Popular posts from this blog

Domain centric architecture : Data driven business process powered by Snowflake Data Sharing

By -
Image
A few month ago I was discussing with the Data Management director of a large manufacturer. We were talking about the architecture principles of having a domain centric data service layer which would feed business processes with harmonised, contextualised and validated data.  In each domain, the data would be owned by domain experts who have extensive knowledge of the data requirements, sources characteristics and usage. Each domain then serves the data "as a service" for it be be consumed and joined with other domains by business processes. This vision provides the ability to quickly drive new use cases and value by leveraging preexisting domains (and related effort) without reinventing the overall data stack in the new "product". However the realisation at scale faces multiple challenges and barriers: currently the data is usually moved multiple times to be prepared and then moved sometimes to each individual product that requires this data, it creates delays, gov
Read more

Snowflake Data sharing is a game changer : Be ready to connect the dots (with a click)

By -
Image
In a previous blog I talked about extended enterprise and how Snowflake can provide the data architecture to support it . Today I will try to illustrate it with a more realistic scenario. The scenario comes from my previous experience in manufacturing but can be adapted to any industry : Several parties need to share their data to get the full picture of a product and then estimate the production costs. This could be part of a Design-to-Cost business process  aiming at optimising product design and fabrication in order to  Use as many as possible off-the-shelf components Reduce raw material usage Reduce the product complexity Mutualise components Reduce waste Eliminate non desired features In order to make it easy to understand I will simplify the data processing part of the scenario and keep the most valuable dimension, the sharing. A composite product Let's imagine we are dealing with a company producing a product and working with several third parties and plants (subsidiaries or
Read more

Process XML, JSON and other sources with XQuery at scale in the Snowflake Data Cloud

By -
Image
In the past few years, I worked with XML and Json technologies in Database. There are still tons of applications especially in the Manufacturing or Finance where XML is leveraged as a data exchange standard. In this new short blog post, the objective is to create a function in Snowflake able to take data from XML, JSON (coming from a VARIANT type) or even from a standard table, view or Snowflake Share and apply an arbitrary XQuery to generate an output in XML (or any other output format thanks to XQuery 3). This function is of course running in the Snowflake compute layer and take advantage of all Snowflake capabilities. The dependencies First we have to load the JAVA dependencies into the Snowflake stage. For this illustration, we again use the user stage. put   file : /// Users / xxxxx / Downloads / saxon-he-test-10.5.jar  @~/ javalib /        auto_compress  =   false     overwrite = true ;                                             put   file : /// Users / xxxxx / Downloads / saxon
Read more