Informatica IDMC Match and Merge Process: A Comprehensive Guide

The Match and Merge process in Informatica Intelligent Data Management Cloud (IDMC) plays a critical role in Master Data Management (MDM) by unifying and consolidating duplicate records to create a “golden record” or a single, authoritative view of the data. This functionality is particularly important for Customer 360 applications, but it also extends to other domains like product, supplier, and financial data.

In this article, we’ll break down the core concepts, the configuration details, and the Cloud Application Integration processes involved in implementing Match and Merge within Informatica IDMC.

1. Key Concepts in Match and Merge

a. Match Process:

• Matching refers to identifying duplicate or similar records in your data set. It uses a combination of deterministic (exact match) and probabilistic (fuzzy match) algorithms to compare records based on pre-configured matching rules.

• The process involves evaluating multiple attributes (such as name, email, address) and calculating a “match score” to determine if two or more records are duplicates.

• Match Rule: A match rule is a set of criteria used to identify duplicates. These rules consist of one or more conditions that define how specific fields (attributes) are compared.

• Match Path: When matching hierarchical or relational data (like customer with their addresses), the match path defines how related records are considered for matching.

b. Merge Process:

• Merging involves consolidating the matched records into a single record. This process is guided by survivorship rules that determine which data elements to keep from the duplicate records.

• The goal is to create a golden record, which is an authoritative version of the data that represents the most accurate, complete, and up-to-date information.

c. Survivorship Rules:

• Survivorship rules govern how to prioritize values from different duplicate records when merging. They can be configured to pick values based on data quality, recency, completeness, or by source system hierarchy.

• Common strategies include: most recent value, most complete value, best source, or custom rules.

d. Consolidation Indicator:

• A flag or status in the IDMC system that indicates whether a record is a consolidated master record or if it is a duplicate that has been merged into a golden record.

2. Configuration of Match and Merge in Informatica IDMC

To configure Match and Merge in Informatica IDMC, there are several steps that involve setting up match rules, survivorship strategies, and managing workflows in the cloud interface.

a. Creating Match Rules

Match rules are at the core of the matching process and determine how potential duplicates are identified. In IDMC, these rules can be created and configured through the Business 360 Console interface.

• Exact Match Rules: These rules compare records using a simple “equals” condition. For instance, an exact match rule could check if the first name and last name fields are identical in two records.

• Fuzzy Match Rules: Fuzzy match rules, often based on probabilistic algorithms, allow for minor variations in the data (e.g., typos, abbreviations). These are ideal for matching names or addresses where slight inconsistencies are common.

• Algorithms like Levenshtein distance, Soundex, or Double Metaphone can be used.

• Weighted Matching: For more sophisticated matching, each field can be assigned a weight, indicating its importance in determining a match. For example, an email match might have more weight than a phone number match.

• Thresholds: A match rule also defines a threshold score, which determines the cutoff point for when two records should be considered a match. If the total match score exceeds the threshold, the records are considered potential duplicates.

b. Configuring Survivorship Rules

Survivorship rules are essential for determining which values will be retained when records are merged.

• Most Recent: Retain values from the record with the most recent update.

• Most Complete: Choose values from the record that has the most complete set of information (fewest nulls or missing fields).

• Source-based: Give preference to certain systems of record (e.g., CRM system over a marketing database).

• Custom Rules: Custom survivorship logic can be defined using scripts or expression languages to meet specific business needs.

c. Defining Merge Strategies

• The merge strategy defines how records are consolidated once a match is identified. This could be a hard merge (where duplicate records are permanently deleted and only the golden record remains) or a soft merge (where records are logically linked, but both are retained for audit and tracking purposes).

3. Cloud Application Integration in Match and Merge

In Informatica IDMC, Cloud Application Integration (CAI) is used to automate and orchestrate the match and merge processes. Cloud Application Integration allows you to create sophisticated workflows for real-time, event-driven, or batch-driven match and merge operations.

a. Key Components of CAI

• Processes and Services: CAI provides prebuilt processes or custom-built processes that handle events (e.g., new records created) and trigger match and merge jobs.

• Business Process Management: You can orchestrate the entire customer data flow by using CAI to manage how and when records are matched and merged based on predefined criteria or user input.

• Real-Time Integration: CAI supports real-time matching, where data coming in from different systems (e.g., CRM, e-commerce platforms) is automatically deduplicated and consolidated into the master record as soon as it is ingested into IDMC.

b. Steps for Cloud Application Integration

1. Triggering Match Process: CAI workflows can be set up to initiate the match process when new data is imported, updated, or synchronized from external sources. For example, a batch of customer records from a CRM system can trigger the match job.

2. Handling Match Results: Once potential matches are identified, CAI workflows can determine whether to automatically merge the records or send them for manual review.

3. Merge Execution: If the match job identifies duplicate records, CAI can trigger a merge process based on predefined merge strategies and survivorship rules.

4. Data Stewardship Involvement: In more complex scenarios, CAI can notify data stewards when manual intervention is required (e.g., for borderline matches that need human review).

c. Automating Matching and Merging with Real-Time Updates

CAI can integrate with external systems using connectors to keep master data up to date across different environments. For example:

• New customer records from an e-commerce platform can be automatically compared with existing records in IDMC to determine if they represent new customers or duplicates.

• Based on the match results, CAI can trigger a workflow that either updates the master record or adds a new record to the system.

4. Best Practices for Match and Merge in Informatica IDMC

• Define Clear Match Rules: Start with exact match rules for critical fields (such as customer ID) and add fuzzy rules for fields prone to variations (e.g., name and address).

• Test Match Thresholds: Experiment with match scores and thresholds to fine-tune the balance between over-merging (false positives) and under-merging (false negatives).

• Monitor Performance: Match and merge operations can be resource-intensive, especially with large datasets. Use IDMC’s built-in monitoring tools to track the performance and optimize configurations.

• Data Stewardship: Set up workflows that allow data stewards to review borderline cases or suspicious matches to ensure high data quality.

The Match and Merge process in Informatica IDMC provides a robust framework for deduplicating and consolidating customer data, ensuring that organizations can achieve a 360-degree view of their customers. However, to get the most value from this functionality, it’s essential to configure match rules, survivorship logic, and cloud workflows thoughtfully. By leveraging Informatica IDMC’s Cloud Application Integration features, organizations can automate and streamline their data unification processes while ensuring high-quality, reliable, and accurate customer records.

Learn more about Informatica IDMC - Customer 360 here