Table of Contents

1. Overview

2. Copyright Notice

The following erwin Metadata Management (EMM) products:

• erwin Web Portal (standard edition)
• erwin Web Portal for Data Governance

are licensed under the following erwin copyright:

Copyright © 2023 erwin, Inc.
All Rights Reserved.

The erwin logo and erwin product names referenced herein are either registered trademarks or trademarks of erwin, Inc. or one of its subsidiaries. All trademarks, trade names, service marks and logos referenced herein belong to their respective companies.

erwin® is a registered trademark of erwin, Inc.

These erwin Web Portal products are based (re-branded OEM) on the following Meta Integration products:

• Meta Integration® Metadata Management (MIMM)
• Meta Integration® Model Bridge (MIMB)
• Meta Integration® Repository (MIR)

which are licensed under the following Meta Integration copyright:

Copyright © Meta Integration Technology, Inc. 1997-2023.
All Rights Reserved.
Meta Integration® is a registered trademark of Meta Integration Technology, Inc.
Other product and company names (or logos) mentioned herein may be the trademarks of their respective owners.
http://www.metaintegration.com

3. Release Changes

EMM 12.0 based on MIMM OEM 11.0.0 (01/31/2022)

• NEW METAMODEL MANAGEMENT FOR IMPORTED "TECHNICAL" MODEL EXTENSIONS
  Imported "Technical" Models are based upon predefined metamodels associated with MIMB bridges/connectors for virtually any data store (file system / object stores / data lakes, RDBMS, NoSQL, DW), Data Integration (DI) and Business Intelligence (BI) tools and technologies. Such predefined technical metamodels can now be extended for data documentation purpose with the same Manage Metamodel (admin) UI used to define custom "business" models.
  Therefore, the new Manage Metamodel (admin) UI not only allows the creation of new custom "business" objects (for the new custom "business" models), but also the creation of new imported "technical" objects defined (scoped) as a set technical objects predefined in the import bridge metamodels. For example a new generic "data field" imported object can be defined as either a RDBMS table column, a NoSQL JSON field, CSV field. etc.
  Consequently:
  • Custom Attributes can now be defined and applied the same way (and therefore reusable) for both imported "technical" objects and custom "business" objects. Not only this eliminates the previous Manage Custom Attributes (admin) UI, but it more importantly avoids redefining the scope of each custom attribute applying to similar imported objects (as it was frequently the case for table/file/entity or for column/field/attribute). help.
  • This allowed the implementation of the term classification process (Now called term documentation) with an actual custom relationship "Defines" from "Term" to a new predefined "Imported Object" in the predefined standard metamodel. Although this new term documentation implementation as a relationship has no impact or direct benefits to the user experience, it offers solutions to the continuous changes in technology and architectures. For example, the data documentation (including term documentation) of a well documented data warehouse on prem (e.g. Teradata) can be exported and reimported to a new implementation of that same data warehouse on cloud (e.g. snowflake).
• IMPROVED DATA DOCUMENTATION AUTOMATION AND PRODUCTIVITY
  The data documentation process of imported "technical" models is a critical part of any data catalog. Any imported object (e.g. tables/files, columns/fields) comes with a physical name which needs documentation with at user friendly (logical) name and description which are now better presented and managed in 3 categories: See help.
  • Business Documentation offers a local documentation with a business name and business description. This can be used as an alternative of the term documentation below, or a mean to supersede an existing term documentation with a better local definition.
  • Term Documentation (previously called term classification) allows to document any imported object with one or more terms from a glossary (now creating a "Is Defined By" relationship)
  • Mapped Documentation allows to document any imported object connected by a semantic mapping with one or more terms from a glossary, or entities/attributes from a data model.
  • Inferred Documentation provides data documentation on any imported object automatically generated from other objects involved its data flow pass-through lineage and impact. This is a powerful feature dramatically increasing the automatic data documentation coverage on many data stores (ODS, data lake, DW) of the Enterprise Architecture.
  The data documentation process and presentation has been considerably improved in the UI (on any imported object overview tab) with wizards for business documentation and/or term documentation (with term reuse or creation on the fly) suggesting business friendly logical names (from naming standards and supervised learning when enabled) from physical names, and description from inferred documentation.
  New "Term Documentation" and "Inferred Documentation" attributes are available in the REST API, MQL, and therefore worksheets and dashboards allowing to create KPI graphical widgets on the data documentation coverage.
• NEW OBJECT CONDITIONAL LABELS
  
  • A new Manage Conditional Labels (admin) UI has been created to define conditional labels based on the Metadata Query Language (MQL) such as "Highly Commented" based on objects with over 10 comments. See help. Each conditional label has a name and icon that can be be searched from an expansive bundled library of icons, customized (e.g. change color), uploaded (from external sources), or even designed in the UI (start from shape, color, etc.). See help.
  • Conditional labels are visible in the overview page of any object. See help.
  • Conditional labels can be displayed in search results and worksheets. See help.
  • Conditional labels can be displayed in data flow lineage diagrams.
• NEW OBJECT WATCHER AND EMAIL NOTICATION
  
  • The Manage Emails (Admin) UI (used to setup notification emails) has been extended to enable the Watcher capabilities at the server level (with an adjustable frequency where the server will check every 15 minutes by default but can be set to hourly or more to avoid loading too much the server). See help.
  • The Manage Object Roles (Admin) UI has been extended with a Watcher Editor capability (allowing a user to start/stop watching an object), and a Watcher Manager capability (allowing to add/remove anyone as a watcher of an object) See help.
  • The Manage Users (admin) UI has been extended to set the watcher notification frequency (of a given user) from daily (default), never (i.e. turned off), to near real time (as setup at the server level). Note that the same watcher frequency can also be setup by each individual user on their top right menu for user profile / preferences. See help.
  • With all above Watcher capabilities configured, users can now see a watcher icon ("eye") on the object overview page at the top right (next to the endorsement icons and menus). The watcher icon shows the count of watchers on that object and offers menus for the user to start/stop watching that object, and possibly add/remove other user watchers if authorized (with with Watcher Manager capability). See help. A new "Watchers" attribute is available that can be used on search, MQL, or for UI customization. See help.
  • The Watcher capabilities are supported on both imported "technical" models and custom "business" models. However, the watcher capabilities are available at the model level only (e.g. not down to just a column or a term object). In case of imported "technical" models harvested as multi-model, one can watch the entire multi-model (e.g. entire database server), or individually watch any desired sub-model such as a given schema of PostgreSQL or a given Workbook of Tableau. See help.
  • Watchers of imported "technical" models receive a separate email per model and per type of activity as follows: See help.
    • Any metadata harvesting driven changes at any level (e.g. add/delete/update of any schema/table/column/type) as soon as (in real time) an import (incremental harvesting) is successful with changes, or failed. In such case, the watcher notification email includes a change summary statistics (e.g. number of added, deleted, updated objects), and a MM server URL link to its model version comparator report for full details.
    • Any other changes such as data documentation (e.g. business name, description, or term classification), social curation, etc. at any lower level (e.g. table, column, data type) as often as defined by the server or a user. In such case, the watcher notification email includes a change summary statistics (e.g. number of changed objects), the top 5 changed objects (with a MM server URL link to the overview page of that object), and finally the detailed changes (with a MM server URL link to the search UI filtered by the content of that model and ordered by last modified),
  • Watchers of custom "business" models, Data Mapping, Semantic Mapping, Physical Data Models receive a separate email per model on any change at any level. See help.
    • Any changes at any level (e.g. add/delete objects, update attributes, add/delete relationships, etc.) as often as defined by the server or a user. In such case, the watcher notification email includes a change summary statistics (e.g. number of changed objects), the top 5 changed objects (with a MM server URL link to the overview page of that object), and finally the detailed changes (with a MM server URL link to the search UI filtered by the content of that model and ordered by last modified),
  • Independently of the above watcher capabilities, other notification emails are also sent to users based on his/her roles/capabilities, including: See help.
    • Workflow transitions on objects where the user has a workflow role, as often as defined by the server or a user
    • Configuration changes (e.g. add/remove model, edit connections) as often as defined by the server or a user, or build errors (in real time) on configurations where the user is a repository manager of that configuration.
    • Server errors (e.g. server down) in real time to the user with Application Administration capability.
• NEW OBJECT ROLES & GLOBAL ROLES
  
  • User roles are no longer pre-defined or hard coded, but instead custom built upon an extensive set of elementary capabilities, such as metadata viewing, data administration, workflow editing, etc.
    User roles and capabilities are either:
    • Object Roles: such as a Curator may have the capability to Comment, Warn or Endorse the metadata in contents to which they are assigned. See help.
    • Global Roles: such as a Security Administrator could have the capability to update group and user role and object assignments. See help.
  • Administrators can associate users or group of users with object or global roles, this association is referred to as a responsibility. This way, one may quickly assign roles and those associated responsibilities to individual users or entire groups of users, as needed. See help.
  • Data governance, data cataloging, data administration, metadata management, data quality, etc., have a variety of "flavors", best practices, roles and responsibilities. Nevertheless, MM is pre-populated with a set of essential roles (such as Content custodian, Data Owner, etc.) that can then be customized. Because MM is so flexible in implementation and nearly infinitely customizable, one may tweak or even re-engineer the groups and roles to fit the very specific needs of a given organization.
    • A set of global roles to provide high-level administration of the metadata management environment. See help.
    • A set of commonly used object roles to allow assignment of all of the capabilities available in the product. See help.
    • More specific global and object roles tailored to specific metadata management activities, scenarios and use cases as identified in the user guide.
    • Specific global and object roles tailored to specific modeled business processes.
    • A RACI (Responsible, Accountable, Consulted, Informed) based example of global and object roles and their assignment.
• NEW WORKSHEET ATTRIBUTES
  
  • The "Stewards" attribute have been moved to the new concept of roles. Therefore, Stewards has been moved from the attribute sheet widget of the Overview tab to a new dedicated Responsibilities tab. Note that that a new widget for Responsibilities is also available for anyone to add in the Overview tab if desired.
  • The "Used" attribute has been renamed to "Has Semantic Usage". More new lineage attributes are also available: "Has Semantic Definition", "Has Data Lineage", and "Has Data Impact". This allows to detect unused objects. All these attributes can also be used as filters.
  • The "Semantic Types" attribute has been renamed "Data Classifications".
  • The "Inferred Semantic Types" widget of the Overview tab has been moved to a new attribute "Data Classifications Matched". More new data classification attributes are also available: "Data Classification Rejected", "Data Classification Approved".
  • The "Term" attribute has been renamed "Is Defined By" and is now a list of Terms instead of one.
  • A new "Term Documentation" attribute shows the list of terms (name and decription) documenting the object (it can also be used as a filter).
  • A new "Mapped Documentation" attribute shows the list of semantically mapped objects (name and decription) documenting the object (it can also be used as a filter).
  • A new "Inferred Documentation" attribute shows the list of terms (name and decription) indirectly documenting the objects through its pass-through data lineage / impact (it can also be used as a filter).
  • A new "Documentation" attribute shows the summarized documentation of the object. The summarized documentation returns the first documentation found on the object following the following priority: Business Documentation > Term Documentation > Mapped Documentation > Inferred Documentation > Imported (Documentation) > Searched (Documentation). This attribute can also be used as a filter.
  • "Business Name Inferred", "Business Name Inferred Origin", "Business Description Inferred", "Business Description Inferred Origin" attributes have been deprecated (but still available in this release) as they have been replaced by the new "Documentation" attributes.
  • The "Documentation" attribute of glossary terms has been renamed to "Long Description" to not conflict with the new "Documentation" attribute described above.
  • New "Data Profiling" attributes have been added "Data Profiling"."Distinct", ."Duplicate", ."Empty", ."Valid", ."Invalid", ."Min", ."Max", ."Mean", ."Variance", ."Median", ."Lower Quantile", ."Upper Quantile", ."Avg Length", ."Min Length", ."Data Profiling", ."Max Length", ."Inferred Data Types"
  • The "Certifications", "Endorsements", "Comments", "Warnings" attributes have been renamed to "Certified By", "Endorsed By", "Commented By", "Warned By". In addition to previously supporting filtering, they can now be used as columns showing the list of users that "Certified", "Endorsed", "Commented" or "Warned" the object
  • The "Endorsement Count", "Comment Count", "Warning Count" attributes have been added to the list of possible filters, allowing to produce worksheets/dashboards with popular objects and more.
  • The "Certified" attribute was added to the list of filters, again for data governance worksheets/dashboards.
  • The "Parent Object Name" and "Parent Object Type" attributes have been added.
  • Object roles can be used as columns or filters
    - filter example: expandedMembersOfRole('Steward') = ANY('Business Users')
    - select example: membersOfRole('Steward')
  • Object relationships/children can be used as columns.
  • Term's workflow "Status" and "State" attributes have been changed into the different and more generic attributes "Workflow State", "Workflow Published", and "Workflow Deprecation Requested" that now apply to any object of a user model under workflow.
  • The "Last Modified Date" and "Last Viewed Date" attributes have been renamed to "Updated Date" and "Viewed Date"
  • The "Created Date", "Created By", "Updated By" attributes have been added (also available as filters). Note: "Created Date" and "Created By" attributes only apply to non-imported objects
• NEW WORKSHEET FEATURES
  
  • In addition to sorting by Name and Relevance, new ability to sort by "Updated Date" in search, worksheets and object explorer (ORDER BY "Updated Date" in MQL)
• NEW CLOUD IDENTITIES MANAGEMENT FOR METADATA HARVESTING
  New MIMB infrastructure allowing password parameters to be based on external (MM managed) cloud identity (on Amazon Web Services, Google Cloud, or Microsoft Azure) where the Secret / Password parameter can be:
  • A secret identifier which is a URL to a cloud identity secret vault's actual secret (allowing for external storage of such secret / password in a cloud secret vault).
  • Empty (no longer mandatory) and the authentication is based on the cloud identity on select bridges (such as Microsoft Azure Data Lake Storage, Microsoft Azure Blob Storage, and more to come).
  Note: early releases provided this feature as MANAGE > Secret Vault. See upgrade details below.
• IMPROVED METADATA REPORTING AND PRESENTATION
  
  • with new graphical widgets (e.g. Responsibilities, Relationships) on object page presentations (e.g. Overview).
    See help.
  • Manage Default Presentation now supports import/export between servers.
• IMPROVED METADATA QUERY LANGUAGE (MQL)
  no longer needing the special character syntax on attributes, and added support for a lot more "system" objects related to data sampling, data profiling, data classification, user roles, and workflow actions dramatically extending the power of the metadata reporting (worksheets and dashboards), and even superseding the hard coded implementation of menus like My Workflow Actions, or My Term Changes now renamed My Changed Objects which is full customizable with MQL.
  For more details on the MQL changes, see the MQL new or improved features, deprecated features, and removed features.
• IMPROVED REST API
  especially in support of the new features of this version such as classification, object and global roles.
  For more details on the MQL changes, see the new or improved API methods, deprecated API methods, and removed API methods.
• IMPROVED ADMINISTRATION UI
  such as Manage Users, Groups, Roles. Classification for a more harmonized and intuitive look & feel and more improved editing capabilities. removed API methods.
• CHANGES FROM PREVIOUS VERSIONS
  
  • Custom Attributes are now defined within Manage Metamodel.
    • All licenses allow to edit custom attributes within Manage Metamodel, however defining new custom objects, relationships and models require a specific license.
    • The repository database upgrade includes a "Migrate Custom Attributes to Metamodel" operation (see Manage Operations for log).
  • Glossaries are now defined within Manage Metamodel.
    • The "Standard" package includes a new Glossary metamodel with new KPI and Acronym objects, but removes the Category object which was the only (previous version) way to create a Term hierarchy. Terms can now contain Terms, and no longer require the creation of a Category.
    • The repository database upgrade includes a "Migrate Glossaries to Metamodel" operation (see Manage Operations for log).
      • The (previous version) Glossary Categories are now migrated as Terms without loss of the hierarchy at the instance level (i.e. glossary term path). Any MQL use of Categories has been updated to Terms as part of the migration. Note that count of Terms might be higher as Categories are Terms.
      • The (previous version) Glossary Term predefined (hard coded) relationships (e.g. More General/More Specific, Contains/Contained by, Represents/Represented by, References/Referenced by, See Also) are migrated as part of an optional "Glossary Extension (MM)" package without loss. However, these relationships are not a mandatory part of the "Standard" package, as users can now define better custom relationships with appropriate names.
      • The (previous version) Glossary Term had 2 workflow related properties:
        • Status has been renamed "Workflow State" with the same values (Candidate, Draft, Under Review, Pending Approval, Approved, Published, and Deprecated).
        • State (Deprecated, New, and Published) is replaced by "Workflow Published" (True/False) and "Workflow Deprecation Requested" (True/False).
      • The (previous version) Glossary Term Abbreviation attribute are migrated to a new "Has Acronym" relationship to a new "Acronym" object. Note that Alternative Abbreviation is not migrated.
      • The (previous version) Glossary Categories and Terms were used to implement naming standards, which are now implemented by a "Naming Standards" model which contains "Naming Standard" objects containing "Naming" objects. When enabling naming standards, users now have to select which "Naming Standard" object they want to use (instead of a glossary category). The repository database upgrade includes a "Migrate Glossaries to Metamodel" operation which also migrate any Categories and Terms used for naming standards into new objects of the "Naming Standards" model.
  • The metadata harvesting browse path (in import/export bridge parameters) is no longer defined as * by default (allowing to browse any drives, directories and files) for obvious security vulnerability reasons. Administrators must use the Setup UI or command line to define the scope of file browsing.
  • The REST API help (MMDoc.war) and any other Tomcat Web Apps are no longer enabled by default for security vulnerability reasons (Swagger unauthenticated sensitive endpoints). They have been moved from $MM_HOME/tomcat/webapps to $MM_HOME/tomcat/dev. If desired, these webpass can be enabled with the Setup UI or command line as follows $MM_HOME/Setup.sh -we mmdoc. This will create the context MMDoc.xml in $MM_HOME/tomcat/MetaIntegration/localhost to make the webapp available and start it.
  • The dashboards can no longer store and execute user defined java scripts for security vulnerability reasons (to mitigate the XSS vulnerabilities). Consequently, a new third party (DOM Purify) strips all XSS properties when rendering the custom HTML of the HTML widget. Therefore, if the HTML contains any tags that are not purely formatting tags (font color, size, images,…), then they will be removed automatically before displaying the HTML. In addition, all HTML attributes allowing to enter javascript (onclick, onload,…) are also be removed.
• THIRD PARTY SOFTWARE UPDATES
  All third party & open source software has been upgraded to their latest versions for bug fixes, improvements, and better security vulnerability protection. For more details, see the published MIMM's Third Party & Open Source Software Usage and LICENSES.
• SECURITY VULENRABILITY UPDATES
  Numerous major improvements to resolve any new security vulnerabilities.
• PRE UPGRADE REQUIREMENTS
  
  • Successful cleanup of the repository See help.
  • Successful upgrade to the previous major release, including the post upgrade manual migration of any single-model database import (deprecated in previous release, now not supported by still working) into multi-model database import. For more details, see the POST UPGRADE section of the previous version release notes.
  • Successful update to the latest MIMM and MIMB cumulative patches for the previous major release for the main EMM server, as well as all metadata harvesting servers. Post patching best practice assumes:
    • successfully re-harvesting (import) of all models (in order not to blame the new major release later),
    • rebuilding of all configurations,
    • deleting any unused versions of models and configurations (it may 3 days for the database to purge deleted models),
    • and making sure that the database maintenance and search index are up to date.
  • Successful database backup and restore of the EMM repository using the actual database restore/backup technology (do not use the EMM application backup).
  • Successful backup of the install data directory.
  • Clean install of the latest build of the full EMM in a new directory (do not reuse/overwrite the previous version install directory)
  • Repository on PostgreSQL Database:
    
    • If you were using the bundled PostgreSQL database server which is only available Windows version of EMM, then this database first need to be upgraded to the new PostgreSQL 13.2 bundled with this new version of EMM:
      (See Application Server Upgrade > Reconfigure your EMM Database Server (ONLY if you are using the bundled PostgreSQL database on Windows).
  • WARNING: Finally, it is highly recommended to first test in a completely separate QA environment configured as follows:
    • Assuming all above steps have been performed in the production environment, make a full (dump) copy of the EMM repository database instance into a new one (it is recommended to temporarily stop the EMM application server and database server for that)
    • Perform a clean install of the full EMM software on an empty directory of the new QA machine.
    • Copy the data directory from the EMM production installation directory to the QA installation directory (this will avoid a full lucene re-indexing)
    • Use the setup utility to point to the new QA repository database instance, and start the EMM application server.
• POST UPGRADE ACTIONS
  
  • Level 1 - Review the logs for any automatic upgrade migration errors
    • MANAGE > System Log

      should look as follows in a typical successful upgrade migration:
   MIRWEB_I0044 Starting database upgrade. Product version is 31.31.2 whereas database version is 30.22.2.
   DBUTL_I0031 Updating database from version 30.22.2
   ...
   MIRWEB_S0005 Running operation: Upgrade data mappings to create classifier links
   MIRWEB_S0005 Running operation: Migrate Custom Attributes to Metamodel
   MIRWEB_S0005 Running operation: Migrate Business Glossaries to Metamodel
   MIRWEB_S0005 Running operation: Migrate Term Classification links to Metamodel
   MIRWEB_S0005 Running operation: Migrate Hide Data property to Sensitivity Label
   MIRWEB_S0005 Running operation: Migrate steward of content with the 'Send email notification when an import' option to watcher
   ...
   SEARCH_I0005 Indexing model [463,1].
   ...
   MIRWEB_S0101 Server is initialized.

      For automated test purpose, the following messages can be searched in the log:

      • On a successful start/upgrade of the server:
MIRWEB_S0101 Server is initialized.
      • On a failure to upgrade the server:
MIRWEB_F0003 Service initialization error:
      • On a failure to start the server:
MIRWEB_F0004 General error during service initialization:

    • MANAGE > Operations

      provides the details for each migration operation (in case of errors on above system log):

      • Upgrade data mappings to create classifier links
      • Migrate Custom Attributes to Metamodel
      • Migrate Business Glossaries to Metamodel
      • Migrate Term Classification links to Metamodel
      • Migrate Hide Data property to Sensitivity Label
      • Migrate steward of content with the 'Send email notification when an import' option to watcher

  • Level 2 - Test Previous Release Basic Features (Object Search, Explore, Edit, Trace Lineage, etc.)
    • Do not start yet any metadata harvesting (model import) and configuration build until you reach level 4 below. In fact, the upgrade process preemptively disables any automatic harvesting post upgrade (see MANAGE > Schedules).
    • Make sure you read the above release notes on: NEW WORKSHEET ATTRIBUTES, and CHANGES FROM PREVIOUS VERSIONS.
  • Level 3 - Test Previous Release Customization Extensions (MQL, Worksheets, Dashboard, Presentations, REST API)
    • Once again, do not start yet any metadata harvesting (model import) and configuration build until you reach level 4 below.
    • Make sure you read the above release notes on: NEW WORKSHEET ATTRIBUTES, IMPROVED METADATA QUERY LANGUAGE (MQL), IMPROVED REST API, and CHANGES FROM PREVIOUS VERSIONS.
  • Level 4 - Test Metadata Harvesting (Model Import and Configuration Build)
    • If you used remote harvesting agents (servers), you must install new ones (based on this version) in a new directory. You may also want to copy the data directory from the old install in order to reuse the metadata cache for incremental harvesting.
      If you need to have both the previous and current version of the software to temporarily coexist on the the same machine for testing (until moving to production), then you must configure the new agents (servers) to run on separate ports (using the setup utility), and update them accordingly on the main server (using MANAGE > Servers).
    • Do not create yet any new model to import with new parameters, instead:
      • It is recommended (but not mandatory) to perform a manual full import (no incremental harvesting), as this new version of the software may bring more detailed metadata. In any case, only import a few selected models to start with (and one at the time). We assume here (per above PRE UPGRADE REQUIREMENTS) that such models imported just fine (with the exact same parameters) in the previous version of the software.
      • A day Later, you can perform more manual full import (one at the time), this time checking the incremental harvesting works.
      • Another day later, you can finally re-enable the scheduled automatic import (metadata harvesting) of that model. Then repeat the above steps again for other models.
  • Level 5 - Test New Release Features from above release notes
  • Level 6 - Patch upgrades from earlier builds of this version
    • "MANAGE > Secret Vaults" capability has been enhanced and replaced by "MANAGE > Cloud Identity".
      The migration should be seamless as the upgrade patch automatically migrates any existing configuration settings for Amazon AWS, Google Cloud, or Microsoft Azure, migrating it all from entries in MANAGE > Secret Vault to cloud identities in MANAGE > Cloud Identity. The ability to use a Cloud Secret Vault to externally store the bridge password parameter is preserved through the migration. Any use of the above secrets vaults as a URL based password parameter of a model import model is automatically detected upon the first (manual or scheduled) import to automatically populate the cloud identity of that model import.
      With this improvement, there is now support for more than one cloud identity per cloud technology. In addition, to support the more robust cloud identity features, select import bridges will support more automatic cloud identity-based authentication. In this way, the password parameter is no longer mandatory for those bridges and the authentication may be based on the cloud identity.

 

EMM 2020 R1 based on MIMM OEM 10.1.0 (GA 01/31/2020 - Deprecated 12/31/2021 - EOL 12/31/2022)

• NEW METADATA REPORTING DASHBOARDS
  with full BI reporting dashboard capabilities (tile layout, widgets for containers, numbers, statistics such as grids, bar or pie charts, etc.) based upon the new Metadata Query Language (MQL) and Worksheets technologies released in the previous version.
  MQL based Metadata Worksheets combined with other dimensions (e.g. history/time, users/groups, configurations, etc.) enable users to produce powerful dashboards (e.g. recently approved terms, what's new, workflow to do list, etc.).
  As with Collections and Worksheets, users have the ability to save/manage/share Dashboards.
• NEW UI CUSTOMIZATION
  based on the above new dashboard technology, allowing users to customize the UI:
  • for each Repository Object (e.g for Terms, Tables, Columns, etc.) where the Overview Tab is now fully editable as a dashboard, and all other Tabs can be individually hidden to simplify the UI for business users.
    Administrators can manage the Default Overview for different metadata presentation to different user groups.
  • for each Configuration Home Page (now available by clicking the banner logo) which is now based on the full power of new dashboards.
    Administrators can manage the Default Dashboard for the home page of different user groups.
• NEW UI LOOK & FEEL
  dramatically improving the user experience:
  • New Header Banner:
    • Simpler customization harmonized between Metadata Explorer and Manager with $MM_HOME/conf/resources/MM.properties.
    • Simpler modern top right menus:
      • Tool submenu with access to other tools (e.g. Metadata Manager) and Help.
      • User submenu with direct access to edit the user account profile (e.g. define full name, email, and now with photo avatar), preferences, and log out.
      • Configuration submenu with direct access to the default, recent and other configurations.
  • New Header Menus with Objects, Collections, WorkSheets, Dashboards, and Manage with harmonized SubMenus to Explore, Manage, and direct access to Favorites and Recent.
  • New Explorer Panel (on the left for navigation) which is now available (harmonized) for Objects, Collections, WorkSheets, and Dashboards.
  • New Object Level Attachments such as associating documents to a Term, Table or even Column.
  • New Object Preview Thumbnail Image manually uploaded by users, or automatically imported by some bridges (such as a BI Report Preview Thumbnail Image when supported by BI import bridges like Tableau and Spotfire).
  • New User Avatar (photo).
  • New User Authentication Management UI harmonized between Native, LDAP, OAuth, and SAML.
  • New End User's Active Operation Monitoring (see below for details)
  • New Administrator's All User Active Operation Monitoring (see below for details)
  • Improved Configuration Management (Build) UI (see below for details)
  • Improved Column/Field Order to physical order by default (instead of alphabetical order) for table columns and file fields (Note that the model must be fully re-harvested to get the new physical order).
  • Improved Data Flow Lineage filtering performance now performed locally on the web client, also reducing the server load.
  • Improved Management UI for Users, Groups, etc. with a harmonized simpler UI.
  • Improved overall layout, graphics (icons) and presentation.
• NEW UI INTERNATIONALIZATION
  with the ability to translate all the UI menus of both the Metadata Explorer and Metadata Manager including:
  • The MIMB Metadata Profile based UI (metamodel vocabulary: e.g. schema, table, column, field, dimension, job, etc.)
  • The MIMB Metadata import/export bridge UI (bridge parameter names, enumeration values, and descriptions/tooltip)
  The internationalization supports automatic detection of the user's web browser local language, including support for multiple languages on connected web clients. However, Internationalization excludes support for multi language content at once (e.g. support for user defined business names and descriptions in both English and French on the same object within a server)
• NEW ACTIVITY MONITORING / OPERATION MANAGEMENT
  with real time monitoring of any activity such as concurrent model imports (metadata harvesting), a configuration build, and other operations such as a repository backup/restore.
  From the UI perspective, a new activity processing icon (spinning gear) appears on the top right of the banner with a counter of concurrent operations. When all operations are completed (no longer active), the gear icon stops spinning. If there was any error in any of the operations, the operation counter is replaced by a red error icon.
  At any time, a user can click on the activity monitoring icon to list the operations, and jump to the desired log, before closing/discarding the activity icon upon completion. Note that only the activities/operations running on behalf of the current user are displayed. In addition, Administrators also have a new Manage/Operations panel to list any active operations running on behalf of any user.
• NEW NEW REPOSITORY MANAGER
  (MANAGE > Repository) has been fully redesigned to replace the legacy Metadata Manager UI.
  Over the past few versions, any new features were developed (and therefore only available) in the modern Metadata Explorer UI. At the same time, all editing capabilities of the legacy Metadata Manager were progressively migrated (while redesigned) into the modern Metadata Explorer UI. The last editing feature was the Semantic Mapper now available in the Metadata Explorer of this new version. All remaining other features (the model version management itself) of the legacy Metadata Manager UI have now been migrated, redesigned and improved as MANAGE > Repository. Links from the Metadata Explorer to Show in Metadata Manager have been replaced by show in Repository Manager. Therefore:
  • The legacy Metadata Manager UI is no longer linked anywhere from the Metadata Explorer UI. New MIMM users should never be exposed to the legacy Metadata Manager UI. The legacy Metadata Manager UI remains bundled in this version as MM/Manager in the URL However, it is deprecated and will be fully removed in the next version.
  • The so called Metadata Explorer UI is now the sole UI as just MM in the URL (instead of MM/Explorer)
• IMPROVED VERSION AND CONFIGURATION MANAGEMENT
  with incremental stitching dramatically accelerating the re-building of configurations based upon the incremental harvesting of multi-models from DI/ETL servers, BI servers (like Tableau), Data Lakes (HFDFS, Amazon S3, etc.), and now large Data Warehouses (Hive, Teradata. Oracle, etc.)
  • New Configuration Build UI preventing concurrent configuration builds, and provide better configuration status updates.
  • New Configuration Model Connection Stitching UI to support the new multi-model databases, including:
    • The 2 levels (Database Model, and Schema) of traditional databases server like Oracle, HIVE.
    • The 3 levels (Database Model, Database Catalog, and Schema) of Microsoft SQL Servers.
  • Improved Configuration Connection UI harmonization between Metadata Explorer and the Manager Manager.
  • Improved Change Management Detection by no longer relying on the physical native id of the object (e.g. table) and instead rely on the name space (e.g. schema/table/column), therefore preventing invalid change detection when the database was re-created (backup/restore), or when pointing from development to production server.
• ARCHITECTURE, DEPLOYMENT & INTEGRATION
  • Improved User Authentication Management (in particular OAuth and SAML) to support the latest version in these standards, and their implementations in popular servers like Azure ADFS.
  • Improved Repository Database Space needed with incremental harvesting of the database servers that are now supported as multi-models at the schema level, as every harvested new version will reuse the stable schema sub-models.
  • All Third Party & Open Source Software has been upgraded to the latest versions for better security vulnerability protection.
• UPGRADE REQUIREMENTS:
  • No specific migration preparation steps are needed as this version upgrade is compatible with the previous one, however:
    • As with previous upgrades make sure you follow proper upgrade steps including applying the current version's latest cumulative patch, making sure the database maintenance is up to date, and a full database backup has been performed.
    • You must perform a clean install of the new software (i.e. this us is not a cumulative patch on top of previous version)
    • If you were using the bundled PostgreSQL database server which is only available Windows version of EMM, then this database first need to be upgraded to the new PostgreSQL 12.1 bundled with this new version of EMM:
      (See Application Server Upgrade > Reconfigure your EMM Database Server (ONLY if you are using the bundled PostgreSQL database on Windows).
• POST UPGRADE:
  • UI Customization:
    Any existing customization of the UI Banner in $MM_HOME/conf/resources/MM.properties or Metadata Explorer in $MM_HOME/conf/resources/MetadataExplorer.xml needs to be manually copied to the newer simpler harmonized version of that file in $MM_HOME/conf/Template/resources.
  • Database Models:
    Large database servers containing multiple schemas (e.g. Oracle, Teradata, etc.) or multiple database instances (e.g. Microsoft SQL Server) are now imported as multiple models (e.g. one model per schema or database instance) with support for incremental harvesting (e.g. only updated or new schemas are imported while unchanged schemas are reused from the harvesting cache). After the upgrade, any newly created database model will be harvested as a multi-model by default. Single large model database import is officially deprecated with this version but still supported, however it will be removed only at the next major release. Therefore, existing database models will go on been harvested as single (large) models for this release only.

    In order to benefit from the new faster and more space efficient multi-model databases, the existing database models can be upgraded using a dedicated operation (Migrate to multi-model databases) which is available at the database model or directory level (in the Repository Manager). This conversion operation converts the single database model into a multi-model database (one model per schema) while taking care of the migration of the database documentation (business names, descriptions, etc.), connection stitching, and any involvement on data flow or semantic mappings. Note that this conversion operation has the following known limitations:

    • When a single-model database content has multiple versions the migration process migrates the latest version only. You can find other versions in the original single-model content moved under the migration folder. These versions will retain relationships to mappings and configurations that use them.
    • Web browser bookmarks of objects in the migrated single-model contents are obsolete as they reference these objects using obsolete identifiers.
    • Migrated contents retain names (paths) of the original contents but not their identifier. For example, it can invalidate references to single-model objects in Worksheet filters
    • Only Diagrams with objects in the same schema are migrated.
    • Configuration Architecture diagrams layout of migrated objects could be lost
    • Migrated Data Mapping will not have previously broken target expressions

 

EMM 2019 R1 based on MIMM OEM 10.0.1 (GA 04/24/2019 - Deprecated 12/31/2020 - EOL 12/31/2021)

• NEW METADATA QUERY LANGUAGE (MQL) & METADATA REPORTING WORKSHEETS:
  • New METADATA QUERY LANGUAGE (MQL)
    allows users to define powerful and complex metadata queries with the familiar SQL syntax. The MQL is available through the REST API and constitute the foundation of a brand new Metadata Reporting UI.
  • New METADATA WORKSHEETS
    based on the new Metadata Query Language (MQL) and a brand new UI. Users start from simple search and powerful filters (automatically building MQL queries) to produce a set of metadata on which the user can select the desired columns (metadata properties) to create/save a Worksheet.
    • Worksheets can be shared with other users (of selected group), and easily managed with quick access to Favorites and search for worksheets shared by other.
    • Worksheets implement the Metadata Tabular Reporting (Grid Mode) solution, and can be switched to simple business friendly reports (List mode).
      Worksheets are also foundations (building block) or the total metadata reporting solution which includes Dashboards in the next version.
    • Worksheets are a lot more powerful than static reports, they allow users to dynamically interact by refining filters, sorting columns, etc.
      In fact, Worksheets also implement the powerful Metadata Bulk Editing capabilities, allowing users to search/filter and easily set some common properties (including custom attributes).
• IMPROVED USER INTERFACE EXPERIENCE:
  • New CUSTOMIZABLE HOME PAGE
    based on an open source sample of a tile based home page with customizable JSON files with short cut icons with internal or external URL, MQL based reports, etc.
    Warning: this custom code home page approach is only a short term solution as it will be replaced by the new official full Dashboard capabilities of the next version.
  • New METADATA EXPLORER MENUS
    have been redesigned to replace the BROWSE menu based on the new metadata reporting capabilities (now with worksheets and soon dashboards) where all existing capabilities (and new ones) have been organized as follows:
    • HOME
      has been removed from the menu bar (to gain space), and replaced by clicking on the top left home logo.
    • A new EXPLORER PANEL
      is available on the left side as a pull down arrow menu (the explorer panel can be pined) which emphasizes navigation using a various organizational structures presented as a hierarchy (tree or drill down). The Explorer Panel offers quick access to metadata through:
      • The Browse tab offers tree navigation from the root of the repository deep into one or multiple models on a single simple tree. It also supports tree browsing from the technical perspective organization of the entire repository, or from the business driven organization of a given configuration. The tree navigation can be set to drill navigation when you need to work with a long list of siblings (e.g., 10K of tables in a database schema).
      • The Search tab offers quick access to metadata query/filters and even save them as reporting worksheets (see above).
      • The Collections tab offers quick access to predefined metadata Collections (see below).
    • OBJECTS
      allows users to search for metadata objects, manage (view, build, favorites, share with me) metadata object collections, look at recently accessed objects (history), browse by object categories (e.g. databases, files).
    • WORKSHEETS
      allows users to manage (view, design, favorites, share with me) metadata reporting worksheets, look at recently accessed worksheets (history), quick access to favorites, dynamically create worksheets by categories and sub categories (e.g. databases / tables / columns)
    • DASHBOARDS
      will be available as part of the next version.
    • MANAGE
      remains unchanged (for administrators only).
  • New METADATA COLLECTIONS
    re-branding the previous concept of Lists and harmonizing it with the new Worksheets in terms of UI usage and management.
  • Improved GRAPHICAL LINEAGE DIAGRAMS
    with better layout for improved object readability and data flow navigation.
• ARCHITECTURE, DEPLOYMENT & INTEGRATION
  • New SSO Authentication with SAML 2.0
    has been added (in addition to OAuth 2.0 added in the previous version). SAML (Security Assertion Mark-up Language) is an umbrella standard that covers federation, identity management and single sign-on (SSO). In contrast, the OAuth (Open Authorization) is a standard for, color me not surprised, authorization of resources. Unlike SAML, it does not deal with authentication.
  • Oracle JDK has been replaced by OpenJDK 11.
  • All Third Party & Open Source Software has been upgraded to the latest versions for better security vulnerability protection.
• UPGRADE:
  • There are no upgrade specific steps as this is a minor version upgrade fully compatible with the previous repository. However, you must perform a clean install of the new software (i.e. this us is not a cumulative patch on top of previous version)
  • Some java bridges depend on the tool's SDK which is not necessarily compatible with OpenJDK 11 which is now the default JRE for the bridges (typically these tool's older SDK can only work with older versions such as Java 8). In such case the bridge tries to automatically run with the JRE bundled with the tool software / SDK instead. When this is not possible, the bridge offers the ability to manually point to a compatible JRE in the bridge's Miscellaneous parameter. Bridges for tools having compatibility issues with OpenJDK 11, include at least Oracle Data Integration, and SAP BusinessObjects.

 

EMM 2018 R1 based on MIMM OEM 10.0.0 (GA 09/21/2018 - Deprecated 09/21/2020 - EOL 12/31/2020)

• BRAND NEW USER INTERFACE EXPERIENCE:
  • METADATA MANAGER VS METADATA EXPLORER UI:
    • In previous EMM versions, the EMM Web Application Server has offered two different User Interfaces (UI) targeting different user communities. The original Metadata Manager UI was designed for the advanced technical users with a traditional development tool layout including multiple panels: tree structure on the left, multi-tab windows in the middle, attributes on the right, and log activities at the bottom. The Metadata Manager UI also presents the highest level of details and complexity of all harvested metadata. The Metadata Explorer UI was initially introduced as a read only UI with simpler metadata for business users offering an easy to use layout for multiple devices, including tablets. The Metadata Explorer became the new UI platform for all new editing capabilities such as the business glossary or data modeling.
    • With EMM v9.8, all other editing capabilities are now available in the Metadata Explorer UI, including data mapping, enterprise data architectures (Configuration editor and model stitching), and even the Administration features like Custom Attributes which are now under are now available in the Metadata Explorer UI > MANAGE > Custom Attributes. Consequently, the Metadata Manager UI is now only necessary (and therefore available) in the EMM Advanced Editions for repository management (with multi version and configuration management). The EMM Standard Edition v9.8 is now fully implemented in the Metadata Explorer UI where MANAGE > Content allows users to directly create models to the default single configuration, import metadata, stitch models (connections), and trace lineage right away.
  • METADATA HOME PAGES:

    New metadata home pages with multiple top tabs offer quick access to all key information:

    • The first tab is always the Overview tab which provides a dashboard to all critical information and properties.
    • The next set of tabs are specific (metamodel / profile driven) to the type of object, for example:
      • Database Table objects have tabs for Columns and Constraints.
      • BI Reports (like Tableau Workbook) objects have tabs for Dashboards, Worksheets, and Data sources.
    • The next set of tabs are for the common critical metadata analysis:
      • DATA FLOW for data lineage and impact analysis.
      • RELATIONSHIPS for detection, management, and curation of relationships (see new features below)
      • SEMANTIC FLOW for definition and usage perspectives (see new features below)
    • The last set of tabs are for common documentation and administration like: Comments, Attachments and Audit Log.

  • METADATA QUICK ACCESS:

    Much improved ways to quickly access the right metadata:

    • SEARCH has been massively improved in both real time performance (now based on Lucene) and in functionality as a metadata driven search with natural language semantic search (see new features below)
    • BROWSE has also been massively improved in both performance (now also Lucene based) and in functionalities as a metadata asset type driven browser with support for hierarchical display at all levels of any data sources including database, DI, BI, Data Lakes, and even No SQL (like JSON hierarchal structures)
    • Enterprise ARCHITECTURE driven graphical navigation allows users to drill down from a top down big picture of the enterprise architecture.

  • METADATA REPORTING:

    Brand new powerful unified metadata reporting capabilities where both search and browse end up to the same reporting page which is also directly available at Browse > Report. Starting from search simply predefines the text filtering (e.g. customer), while browsing predefines a category (e.g. database / tables), and direct access to reporting does not predefine anything.

    • The reporting capabilities offers to select multiple categories (e.g. database / tables + Flat files) and subset by content (My Data lake + Sales DW database) before drilling down with the following filters:
    • Then filtering is available for Last Modified, Stewards, Labels, Semantic Types, Endorsed By, Certified By, Created By, Warning By, and Commented By.
    • Finally, more custom filtering per attribute (including custom attributes) common to the metadata subset (e.g. SecurityLevel = Orange).
    • Reports can be reused by saving the URL as favorites (further versions will support full report management within the application)

  • METADATA USER LISTS:

    Brand new user list management feature allows users to define and mange lists of metadata objects. Just like labels, lists are available anywhere in the UI to add/remove objects, bulk editing, and management. Lists can contain any type of metadata such my favorite list of terms, tables, or reports. Lists can also can contain multiple type of content such as my to do list with terms, tables, and reports in that list. Lists can be shared with other users when marked as public, such as our quarterly review list. Note that lists are flat, therefore not hierarchical and with no sub-list or include concepts.

  • METADATA TAGGING WITH LABELS:

    The metadata tagging with labels has been much improved to be harmonized with the brand-new list management experience in order to facilitate adding/removing objects anywhere, grid editing, and more.

  • METADATA DOCUMENTATION:

    Much improved ways to document metadata:

    • MULTI-LINE TEXT has been introduced (in addition to the previous single line Text for better formatting and layout. In addition, Multi-Line text has been enhanced with support for URL links and embedded image attachments using a JIRA like syntax. Multi-Line Text is not only the default format for all Descriptions and Comments, but is now also available as a new type of Custom Attribute that can be applied to any metadata for documentation.
    • RICH TEXT Documentation with (WYSIWYG) Visual Edition is not only the default medium for Glossary Term documentation, but is now also available as a new type of Custom Attribute that can be applied to any metadata for documentation.
    • SQL TEXT of SQL View, Stored Procedures and more are now better presented with colored syntax and optional reformatting. Note that this is not a new type of custom attribute but any predefined attribute with SQL is better formatted.
    • ATTACHMENTS (such as pictures, documents, etc.) have been enhanced as part of its integration with the new Metadata Explorer, including Management (Drag and Drop), Preview, and Thumbnails that can be embedded in the Text (and Multi-Line text) descriptions, comments and custom attributes.

• DATA MODELING AND DOCUMENTING ANY HARVESTED METADATA
  • In EMM v9.7, existing data stores such as RDBMS could be harvested as a Physical Data Model (PDM) instead of a simple Model, in order to offer full documentation including business glossary term reuse based upon automatic semantic links, reverse engineering based upon naming standards, data modeling with diagramming, and of course automatic change management (re-harvest/compare/merge).
  • In EMM v9.8, all the above capabilities are now available on any harvestable model content without having to create a PDM. In other words, any data integration, business intelligence, reports, data stores (relational, hierarchical, NoSQL, files, etc.) can be documented at as needed, including support for relational data models. Consequently, all existing PDM in EMM v9.7 may be converted to Models in EMM v9.8 without loss of any existing documentation (including diagrams).
  • The documentation (business names and definitions) process has been improved allowing any object (e.g. table, column, report field) to be quickly and easily:
    • "Classified" with a local semantic link to a glossary term, without having to use an intermediate Semantic Mapping content, or associating the Model to a Glossary as with the PDM.
    • "Documented" with a local business name and definition overwriting any Semantic link (Classified, Mapped or Inferred)
    Furthermore, this documentation process is also dramatically enhanced through the integration with a new "Semantic Flow" tab acting as an interactive dashboard on finding the right definitions (see below).
  • When harvested databases that are already documented in data modeling tools (e.g. Erwin), such data models can be imported as a separate model and automatically stitched directly to its matching harvested database (without using any semantic mapping model). The semantic stitching is automatically maintained as both the database and its associated data model are independently re-imported/refreshed on regular basis (the stitching will report inconsistencies). From the user perspective, the documentation (business name, descriptions, relationships, diagrams) of any harvested database table / column is automatically inherited from its associated data model.
  • Note that EMM Advanced Edition with Authoring also allows one to use a PDM to create data models from scratch (e.g. design new HIVE table requirements) without pointing to a live database rather than simply documenting existing data stores. The PDM concept is retained for this purpose. Note that a Conceptual/Logical Data Modeling capability may also be added in future versions.
• DATA CATALOGING
  • Brand new Data Cataloging applications well integrated with the existing Data Governance (DG) capabilities, and based upon the solid Metadata Management (MM) foundations with full data linage and powerful metadata version and configuration management.
  • Managing both modern cloud based data lakes and classic Data Warehouse (DW) Enterprise Architectures
  • Harvesting metadata from both modern (XML, JSON, Avro, Parquet, ORC) files, Hive tables, and Kafaka messages), and classic (relational tables / CSV files) data technologies
• RELATIONSHIPS Discovery & Management
  • Relationship Discovery using the following methods:
    • Automatically "Inferred" based on:
      • Metadata Usage Driven: using the surrounding data flow usage such as joins in DI (ETL Tools, SQL Scripts or Data Prep) and BI (traditional or self-service) activities.
    • On Demand "Detected" based on:
      • Metadata Name Matching: for example PurchaseOrder.SKU = Product.SKU or Customer.AccountId = Account.Id)
      • Semantic Definition Matching: classified by users to the same glossary term.
      • Semantic Type Matching: discovered through data profiling (e.g. SSN syntax or VIN number)
  • Relationship Management with user defined relationships and social curation (e.g. endorsed or certified joins)
  • Dynamic Data Model diagram generation from Relationships surrounding any object (e.g. table or file).
• SOCIAL CURATION
  • Endorsement, warnings, certifications with impact on search ranking.
• SEMANTIC SEARCH
  • Metadata driven search language such as "Customer tables" for any tables with Customer in the name, "tables with SSN" for any table with a SSN column (e.g. for GDPR), or "ROI in Reports" for any reports containing ROI.
• SEMANTIC MAPPING
  • Major improvements in semantic mapping including in place semantic mapping via two approaches:
    • Top-down from Business Glossary Term or Data Model Entity/Attribute,
    • Bottom-up from Data Store Tables/Columns or Report Fields.
• SEMANTIC FLOW
  • Major improvements on the semantic flow analysis now also supporting the documentation process acting as an interactive dashboard for finding definitions that are:
    • "Local" (within the Model) that has been either "Imported" (metadata harvesting) or locally "Documented" (edited description overwrite),
    • locally "Classified" (within the Model) to an external glossary term,
    • directly "Mapped" via a Semantic Mapping Model or direct stitching (e.g. between a database and its data model),
    • indirectly "Inferred" through complex data flow pass through and semantic flow (which can be graphically analyzed in the data flow diagram), or
    • "Searched" for by name in all glossaries.
    Any of the Searched, Inferred or Mapped definitions may quickly (in place) be reused/promoted as a Classified or Mapped definition.
• DATA CONNECTIONS / METADATA STITCHING
  • Complete support for file format harvesting and stitching.
  • Connection pool factorization (e.g. from DI and BI servers) to minimize the number and complexity of stitching connections.
• DATA MAPPING
  • The Data Mapping Specifications and the Data Mapping Designs have been fully resigned and merged into brand new Data Mappings that can be used for multiple purposes, including capturing data flow mapping requirements.
  • The Data Mapping tool allows for the mapping of multiple source data stores into a target data store in multiple steps with (schema or table level) Bulk Mappings and (column/field level) Query Mappings. The data mapping tool offers new graphical mapping visualization as you map, and new expression syntactical editors when designing joins, lookups, filters, etc.
• ARCHITECTURE, DEPLOYMENT & INTEGRATION
  • Search engine re-designed and optimized with Apache Lucene offering near real time metadata search and navigation (and removing any dependencies of underlying database text search requirements)
  • Third party software upgraded to the latest of Java 8, Apache Tomcat 9, PostgreSQL 10 and more for security and performance improvements.
  • Single Sign On (SSO) integration architecture has been redesigned for easy external authentication with redirect using custom scripts in any language such as Python (Note that EMM v9.7 external authentication required custom complex integrated java scripts). This includes new support for SSO Authentication with OAuth 2.0. Post GA cumulative patches will include support for the Security Assertion Markup Language (SAML) standard, and native cloud authentications such as Amazon AWS and Microsoft Azure.
• GENERAL:
  • All Java code is now compiled with OpenJDK.
  • All Third Party & Open Source Software has been upgraded to the latest versions for better security vulnerability protection.
• UPGRADE REQUIREMENTS:
  • CPU & MEMORY:
    The EMM v9.8 now uses Lucene as the search engine instead of using the text search capabilities. Consequently, the overall hardware requirements now demand as much processing resources (CPU and memory) on the EMM Application Server machine as on the underlying Database Server machine. (EMM v9.7 use to demand more resources on the database server as all searches were preformed using the database server).
  • DATABASE:
    The EMM v9.8 server now requires PostgreSQL database software version 10 or newer.
• UPGRADE PROCESS:
  • POSTGRESQL DATABASE VERSION UPGRADE:
    If you deployed your EMM v9.7 server connected to your own PostgreSQL server, then you must first upgrade the PostgreSQL server to version 10 or newer, and make sure the EMM v9.7 software works still properly after that step.
  • EMM v9.7 SOFTWARE UPGRADE & DATABASE MAINTENANCE:
    You must download and apply the latest EMM v9.7 Cumulative patch (July 2018 or newer), and make sure the EMM v9.7 software still works properly after that step.
    You must then make sure that the regularly scheduled database scheduled has ran successfully and long enough to have no further actives to do (such as purging deleted content from the database)
  • EMM v9.7 DATABASE BACKUP:
    You must then perform a full database backup (EMM, data, indexes and scripts), restore it on a brand new (different) database server, and make sure the EMM v9.7 server software still works properly after that step.
    (Do not rely on the EMM backup script for that process, you must use real full database backup).
  • EMM v9.8 SOFTWARE INSTALL & DATABASE UPGRADE:
    Install the EMM v9.8 software on a new directory (independently of the EMM v9.7 install directory). Use the Setup utility to point to the database to upgrade. Starting the new EMM v9.8 server will automatically perform an upgrade of the database's EMM data and scripts. Note that EMM v9.8 uses Lucene on the EMM server for search, therefore the database will no longer need to use text search indexes and will therefore need less space in the database
• POST UPGRADE:
  • FLAT FILES:
    EMM v9.7 had no proper support for flat files which are now fully supported as part of the new data cataloging capabilities of EMM v9.8. Therefore, the early Flat File (CSV or XLSX) prototypes beta bridges have been discontinued and replaced by the import bridge from File System (CSV, Excel, XML, JSON, Avro, Parquet), or the other new file system / object store import bridges such Amazon S3, Azure Blob Storage, Hadoop HDFS, etc. which can all contain flat files. All these file system import bridges now create multi-model content which can be stitched to data mappings and other DI/ETL models. Any content imported from Flat File (CSV or XLSX) prototypes beta bridges can still be visible in EMM v9.8, but no further imports can be performed, and migration to the new bridges cannot be automatized (different parameters and different content type from single to multi model). Therefore, new models should be created with the new file system bridges, and the old content (from the prototype beta bridges) should be deleted.
  • PHYSICAL DATA MODELS (PDM):
    In EMM v9.7, PDM were used to document (including data model diagrams) existing databases, as well as designing new databases (new tables, columns, etc.). In EMM v9.8, PDM is no longer needed to document existing databases (including data model diagrams, glossary integration, etc.). Therefore, PDM is now only needed for data store requirements / database design which requires EMM with "metadata authoring" license. Consequently, EMM servers without "metadata authoring" license" can no longer create new PDM models, although any existing PDM created with EMM v9.7 are still operational on EMM v9.8. The bottom line is that the EMM v9.8 new way to document data stores / databases is not only as powerful as PDM but it is much more efficient with version management (changes after new harvesting). Consequently, we recommend all PDM models used to document databases to be converted to models which can be performed without any loss of documentation, by means of a conversion script available on PDM models.

 

EMM 9.7 based on MIMM OEM v9.1.0 (GA 04/24/2017 - Deprecated 12/31/2018 - EOL 12/31/2019)

• HARVESTED MODEL CHANGE DETECTION avoids systematically creating a new version of the content upon scheduled harvesting of large database, data modeling, data integration, or business intelligence servers. This is automatically performed as side effect of any new harvesting by a systematic and efficient (fast) model comparison with the previously harvested metadata This dramatically reduces the needed database space by creating fewer versions, and enable the possibility of reliable notification only if a change really occurred. Note that in previous versions, the incremental harvesting already offered systematic and automatic reuse of unchanged models (e.g. data model) from a large a multi-model server (e.g. data model repository)
• SUBSCRIPTION / NOTIFICATION mechanism on select changes at both the repository model level (e.g. a new version of a data store is harvested) and at the model object level (e.g. a terms changed or ready to approve/review in a Business Glossary under Workflow). Anyone may be assigned stewardship roles and thus will be notified as new imported content is harvested, with links back to the new object and the ability to compare using the newly re-written powerful comparator, and the ability to identify impacts of change for any architecture or configuration of assets.
• CUSTOM ATTRIBUTES on any harvested metadata (models) from data stores, data modeling, data integration, and business intelligence tools. Therefore allowing to tag any metadata (from Hive tables to Data Models down at the column/field level) for custom properties such as a company confidential level that can be read for external products on security enforcement and compliance. Note that in previous versions, these same custom attributes could already be applied to authored metadata (models) such as terms of Business Glossaries or tables/columns of Data Models.
• BUSINESS GLOSSARY WORKFLOW for the business users behind the Metadata Explorer UI. Note that in previous versions, a (customizable) Workflow was already available in the Metadata Manager.
• DATA INTEGRATION BROWSING AND LINEAGE for the business users behind the Metadata Explorer UI, allowing them to browse the data integration jobs (from DI/ETL/ELT tools as well as SQL Scripts) and analyze the summary data flow lineage of their execution. Note that in previous versions, the browsing (open) and full detailed data flow lineage of any data integration models (DI/ETL/ELT and SQL scripts jobs) were already available in the Metadata Manager (and still are).
• DATA CONNECTION STITCHING MAJOR IMPROVEMENTS now offering to stitch by column position (needed in SQL insert statements), and smart case aware stitching (as some database's name spaces like schema/table/column are case sensitive while others are not)
• BULK DATA MAPPING at the table level.
• MODEL COMPARE/MERGE: The comparison facility has been completely re-written to include comparison for every level of detail for those models with the same profile (e.g., data model from one technology and data model from another). Even entirely different contents (e.g., a data model, or and Glossary) may be compared, only at a lessor level of granularity (basically at the granularity of stitching, e.g., schema, table, column). Finally, for physical data models (including documentable models based upon harvested database structures) one may use a powerful merge feature again with full control down to any level of granularity.
• DATA MODELING Major improvements and new features involving the physical data modeling capabilities centered around two major use cases:
  • A data documentation tool enabled through physical data modeling based upon the structure harvested from existing data stores of the of data warehouse and data lake (including traditional RDBMS and Hadoop Hive big data)
  • A data requirements tool for new data stores to be defined such as self service DI to new Hadoop Hive tables in the data lake.
  Also includes new advanced support for physical data model diagram editing, relationships definition, all the way to export to data modeling tools, business intelligence design, etc., and comparison/compliance with live database / big data hive implementations..
• REST API SDK major enhancements with many new features
• LIGHTWEIGHT MODELS: Model content (such as a harvested databases or data models) can be stored in the repository as a lightweight model (just the XML file), or fully expanded (as both XML file and fine grained repository objects). When retaining many historical versions of a model, using lightweight models saves repository space, and also avoids slowing down the search by not indexing historical repository objects. Lightweight models cannot be directly used in a Configuration or in a Mapping. However, the lightweight model of a data store (such as a RDBMS or Hadoop Hive) can be documented with a Physical Data Model (PDM) for data model diagramming and semantic linking to a Business Glossary (BG). Such a PDM can of course be used in any Mapping or Configuration to be exposed to business users in the Metadata Explorer. Note that lightweight models can immediately (without any loss of performance) be opened in the Metadata Manager (to browse metadata or trace lineage within that model), Compared (with the Model Comparator to analyze the difference between versions), Exported (for example to BI design tool).
• UPGRADE NOTES: The Business Glossary Workflow has been significantly improved in the metadata explorer. Therefore, before upgrading to this version, it is strongly recommended that all business glossaries under workflow have published terms only (this can be achieved by listing all draft terms and either publishing or reverting them to the previously published state). Otherwise, the upgrade process will systematically publish all draft terms in their current state (which can be undesirable if the current state is not what you would like to publish). Finally, after the upgrade, you will need to manually re-enable the Workflow on these Business Glossaries.

 

EMM 9.64.2 based on MIMM OEM 9.0.2 (GA 06/10/2016 - Deprecated 12/31/2017 - EOL 12/31/2018)

• METADATA MANAGER:
  • Major improvements in MODEL VERSION / CHANGE MANAGEMENT preventing the creation of unnecessary new versions of models if the source metadata (e.g. a database, or data model) has not changed since the last automatically scheduled metadata harvesting. This is new feature is achieved by taking advantage of the MIMB's new capabilities to compare the metadata of a newly imported model with a previously imported one in order to detect any change. The major benefit of this new feature is to dramatically reduce the disk space in the repository by automatically deleting unnecessary versions.
  • New CONFIGURATION / VERSION CHANGE MANAGEMENT capabilities offering a comparator of versions of configurations.
• METADATA EXPLORER:
  • Major redesign of the Metadata Explorer UI on both the look & feel and actual capabilities.
  • New METADATA SEARCH/BROWSE with FILTERING AND REPORTING capabilities offering:
    • New Search (or Browse) with Filtering capabilities on any attributes/properties
    • New choice of result display as a classic Google like "LIST", or a new powerful "GRID" offering a spreadsheet like display of multiple attributes/properties at once. Such attributes/properties can be individually selected, reordered, and sorted, basically offering a full metadata reporting solution. The results can of course be downloaded as CSV/Excel files.
  • New METADATA EDITING Capabilities at many levels including:
    • Numerous new fast and easy "in place" editing to Rename objects, Edit Descriptions, and more.
    • The new Search/Browse GRID display also offers efficient editing with:
      • TABULAR EDITING of multiple objects at once such as Business Glossary Terms, Data Models Tables, or Table Columns
      • BULK CHANGE of multiple objects at once, where a search can return multiple objects (that can then be selectively subsetted) for which changes can be performed at once (e.g. Change the Security Threat Level to Orange to a set of tables at once)
• METADATA AUTHORING TOOLS:
  • Common Features:
    • Major improvements in CUSTOM ATTRIBUTES (also known as User Defined Properties) to objects of the Business Glossaries and Data Models
      • Custom Attributes are now common to the EMM repository (e.g. Security Threat Level = [ Red, Orange, Yellow, Blue, Green ] and are therefore shared between Business Glossaries, Physical Data Models, etc. Therefore, having a centralized place for maintenance (e.g. adding a new value: purple)
      • Custom Attributes can now have a default value (e.g. default value is green)
      • Custom Attributes now have a much wider scope to be applied at any level from a high level repository object (e.g. harvested Model, Data Mapping, Directory) to fine grain model objects (e.g. A Business Glossary / Term, and/or a Physical Data Model / Column)
      • Custom Attributes now have a security group associated to them (e.g. the Security Threat Level custom attribute may only be set by a custom Security Approved group)
    • New AUDIT TRAIL for any changes in objects of Business Glossaries and Data Models, including who changed a given attribute and when
  • BUSINESS GLOSSARY:
    • New Business Glossary editing capabilities are now available to the business users under the Metadata Explorer UI. (includes tablet friendly in place editing, as well as HTML formatting of descriptions, etc.)
  • DATA MODELING:
    • Common Features:
      • Major improvements in data modeling DIAGRAM EDITING: entity formatting, relationships editing, automatic layout, etc.
    • PHYSICAL DATA MODEL (data documenter for existing data stores databases, data warehouse, data lake):
      • Major improvements (as part of the above common features)
    • LOGICAL DATA MODEL (brand new feature) (enterprise conceptual/logical information modeler):
      • This new feature is postponed to the next release
  • DATA MAPPING:
    • DATA MAPPING SPECIFICATIONS (Data Flow Mapper)
      • Minor improvements and bug fixes
• ARCHITECTURE & TECHNOLOGY:
  • Major database performance improvements
  • Updated EMM Web Services which are now based on RESTful API technology (i.e. therefore removing any security vulnerabilities of the older Axis technology)

 

EMM 9.64.1 based on MIMM OEM 9.0.1 (12/15/2015)

• ARCHITECTURE & TECHNOLOGY:
  • 100% Java delivery and installation allowing support for Windows as well as a variation Linux/Unix deployments
    • The Metadata Management Server (EMM) can now be installed on Unix/Linux variations.
    • The Metadata Harvesting Agent (MIMB bridges) can be installed:
      • Locally (co-located with a EMM server on Linux) to run 100% java based bridges including JDBC database bridges (Oracle, Teradata, DB2, SQL Server, etc.), big data bridges (Hadoop Hive, HCatalog), and other popular bridges such as ERwin xml, etc.
      • Remotely on a Windows machine for C++ based bridges and COM API based bridges requiring software SDK running on Windows such as ERwin native (.erwin) files.
  • 100% HTML 5 (no more Flash) and tablet friendly look & feel, allowing to run on Mac, Tablets, and more
  • Metadata Explorer Customization to offer a better experience to targeted business users of the customer company:
    • Customize headers, company logos, menus, search categories, home page (with search widgets).
  • Improved Metadata Explorer Search Performance
• METADATA AUTHORING:
  • Improved DATA STORE DOCUMENTER (PHYSICAL DATA MODEL):
    • Live harvesting and automatic update (with version management) from popular RDBMS or big data hadoop data store
  • Common Data Modeling Capabilities:
    • Data governance integration with Business Glossary including naming standards, reuse of terms, creating of terms on the fly with supervised learning
    • Data modeling graphical diagram editor (relationships, annotations, auto layout, etc.)
    • Full integration (import and export) with data modeling tools like ERwin

 

EMM 9.6.1 based on MIMM OEM 8.0.3 (05/19/2015)

• DATA DOCUMENTER (NEW):
  • Full integration (import and export) with data modeling tools like ERwin
• DATA MODEL DIAGRAM VISUALIZER:
  • New diagram auto layout
  • New dynamic layout of a diagram subset starting from an entity with all related entities with one or two levels of relationships (very useful for large diagrams)
• SEMANTIC MAPPER:
  • New support for models within a multi-model server source, allowing to provide documentation (or glossary) at the high level of a given data model within a multi model server, such as:
    • a Data Modeling (DM) repository server with many data models inside (e.g. ERwin Mart)
• BUSINESS GLOSSARY:
  • New customizable role driven workflow support (can be turned off) and security enforcement
• METADATA EXPLORER UI:
  • New integrated presentation of Business Glossary terms related to any data store, including the ability to add/remove BG terms documenting a data store
  • Improved search support for auto complete and objects
• ADMINISTRATION:
  • New group based security model (as side effect of the new role based Business Glossary workflow)

 

EMM 9.6.0 based on MIMM OEM 8.0.2 (03/18/2015)

• ARCHITECTURE & TECHNOLOGY:
  • New support for HTML5 only devices like iPad and other tablets (Flash will no longer be needed) for graphically tracing any data flow or semantic lineage, visualizing data model (Diagram Visualizer), or editing the enterprise architecture layout (Configuration Manager).
  • Improved search performance EMM persistence on SQL Server
  • Java 8 (compiled with backward compatibility with Java 7) compliance (Java 6 is no longer supported)
• METADATA MANAGER UI:
  • New Business Glossary (for Data Governance Edition only)
  • New Data Documenter (for Data Governance Edition only)
  • New Data Flow Mapper (for Data Governance Edition only)
  • New Metadata Manager look & feel (to match the Metadata Explorer)
• METADATA EXPLORER UI:
  • Brand New (redesigned from scratch) Business User Interface with simplified search, navigation and presentation paradigms for business users to easily access and understand the metadata assets
  • Replaces the "Metadata Explorer" UI which was a read only version of the "Metadata Manager" UI with simplified metadata for business users.
  • New customizable action menus per repository object type (e.g. open BI report with BI tool by default)
  • New dedicated web pages for tracing data lineage & impact, and semantic definition & usage
  • New access to the Configuration's Enterprise Architecture Diagram
  • Improved search support for auto complete and objects

 

EMM 9.5.0 based on MIMM OEM 7.2.0 (11/01/2013)

• IMPROVED: Support for SQL Server 2012 Databases
• IMPROVED: Web-Based Architecture Diagramming of Configurations
• IMPROVED: Configuration Management of Mart Subsets
• IMPROVED: Semantic Lineage Following Roll-up, Roll-down and Many-to-Many Transformations
• IMPROVED: Enterprise Authentication (e.g., automatic Windows-based authentication)
• IMPROVED: “High Fidelity” Diagram Visualizer
• IMPROVED: Minor enhancements and bug fixes

 

EMM 9.2.0 based on MIMM OEM 7.1.0 (07/30/2013)

• IMPROVED: “High Fidelity” Diagram Visualizer
• IMPROVED: Enterprise Authentication (e.g., LDAP based authentication)
• IMPROVED: Minor enhancements and bug fixes

 

EMM 9.1.0 based on MIMM OEM 7.1.0 (04/05/2013)

• NEW: “High Fidelity” Diagram Visualizer
  • For “Faithful” rendering of the original model diagrams as developed in ERwin
    • Retains original object positions and sizes
    • Properly supports auto size entities
    • Retains display levels for individual objects and the diagram as a whole
    • Retains original entity fonts
    • Retains original attribute data type alignments
    • Retains other shapes and additional documentation on the diagram
    • Separates Logical and Physical Diagrams
• NEW: Diagram Relationship Analysis allows the user to highlight the PK/FK's involved in a relationship and even generate the relationship join expression which may be cut and pasted for immediate use in SQL queries.
• NEW: Powerful Search Language
  • Allowing for sophisticated / advanced search such as: Any Words, Exact Phrase, All Words, Exclude Words, Wild Card End, Parent and Child, Exact Name, Object Type, Property Type.
• NEW: Model Browsing Experience
  • Offering a separate model browse panel simultaneously displayed adjacent to diagrams and lineage
• IMPROVED: Semantic and Data Flow Lineage & Impact Analysis
  • Direct access to essential analysis:
    • Trace Data Lineage (e.g. views to tables)
    • Trace Data Impact (e.g. tables to views)
    • Trace Semantic Definition (e.g. physical tables to logical entity)
    • Trace Semantic Usage (e.g. logical entity to physical tables)
  • Choice of presentation:
    • Simple List (of ultimate sources and/or destinations)
    • Advanced Graph (of semantic and/or data flow)
  • Choice of direction:
    • Reverse (for Data Lineage or Semantic Definition)
    • Forward (for Data Impact or Semantic Usage)
• IMPROVED: Data Flow Advanced Analysis
  • Separation of data flow and control flow:
    • Data flow (actual movement of data)
    • Control flow (conditions and filters)
      • Column Control (which directly impacts values)
      • Row Control (which does not directly impact values such as filters)
  • Visualization of Data transformation Type of lineage by using color and/or thickness of lines:
    • gray lines means data flow with no transformation (i.e. pass through)
    • black lines means data flow with transformations (e.g. expression)
    • thick lines means data flow with transformation processes (e.g. ETL workflows that can be analyzed when clicking on such lines)
    • yellow lines means column control flow (e.g. lookups)
    • yellow dashed lines means row control flow (e.g. filters)
    • blue lines means semantic flow
• IMPROVED: Search
  • Search improvements for SQL Server wildcard/word break searches

 

EMM 9.0.0.1 based on MIMM OEM 7.0.4 (11/15/2012)

• MAJOR IMPROVEMENTS:
  • New and Improved LDAP support:
    • Integrated Native and LDAP support (by default)
    • Possible use of LDAP for authentication only, while using (the user friendly UI) Native Role Assignments for LDAP Users
    • Possible use of LDAP predefined groups (with a user friendly "search" on LDAP groups) for automatic LDAP driven Role Assignment
    • User friendly UI to create an LDAP query for a virtual group of LDAP Users (e.g. small data administrator group) with automatic LDAP driven Role Assignment (e.g. to Administrator in that case)
  • Updated search results and filter to display model name along with Model Directory name
• CRITICAL BUG FIXES:
  • "Failed to upload License File" error message on "Save License" in the License window to initialize Web Portal Database.
  • Import/Export Log display unreadable (occurred only on IE, and only when connected to a Web portal on top of SQL Express)
• INCREMENTAL PATCHES
  Note: The MITI Support Knowledge base contains information describing updates provided as part of the incremental patch process here.

 

EMM 9.0.0.0 based on MIMM OEM 7.0.4 (10/12/2012)

• NEW FEATURES:
  • METADATA TAGGING/ANNOTATIONS/LABELS: Ability for business users to "tag" any metadata with labels, at any granularity level from models down to a particular column
  • USER COMMENT MANAGEMENT: Ability for business users to "comment" on any metadata, at any granularity level from models down to a particular column. Model Administrators (data stewards) can then review and manage these user comments (e.g. change status from Candidate to Approved). Finally, Model Administrators can push selected comments back into the original tool (e.g. ERwin model)
  • FLOATING LICENSE SUPPORT: offering new concurrent user licensing, with options for one or both of:
    • Named (exclusive) users
    • floating (first come first served) users
• MAJOR IMPROVEMENTS:
  • METADATA PROFILING: major improvements in accurately representing the original tool metamodel
  • LINEAGE TRACING INTERACTIONS: major usability improvements
  • DATA FLOW LINAGE SUMMARY: greatly improved the summary lineage presentation
  • METADATA EXPLORER SIMPLIFICATION: using context dependent tree navigation, therefore limiting the nesting of left navigation panels
  • TOUCH SCREEN TABLET SUPPORT: with full menus available on Action button as a substitute to the right click menu
  • CROSS WINDOW/TAB OBJECT NAVIGATION: with Show Object in Model Browser, in Diagram Visualizer, in Lineage Analyzer, etc.
  • GENERALIZED WEB LINKS AND BOOKMARKES: to any objects in a Model Browser, in a Diagram Visualizer, in a Lineage Analyzer, as well to the precise context of any actions like search and reporting
• INCREMENTAL PATCHES
  Note: The MITI Support Knowledge base contains information describing updates provided as part of the incremental patch process here.

 

EMM 8.2.0.2 based on MIMM OEM 7.0.2 (1/31/2012)

• Initial Release

 

4. System requirements

4.1 Important preliminary disclaimer notice on all requirements

The following requirements only define the minimal requirements to run the application server with reasonable performance based on the provided tutorial, or small business use cases. The actual requirements for enterprise wide use cases based on larger models and configurations do require significantly greater resources to obtain acceptable performance.

The following requirements are based on:

• minimal to no network overhead (assuming both the database and Application servers to be locally installed),
• vendor's default install of the current version of their software (with all current service or fix packs),
• no other applications sharing such hardware (starting from a clean machine),

Any other hardware/software configurations are acceptable as long as they provide the same (or better) results on the provided performance benchmark. In such case, if any problem is discovered (e.g. scalability or performance issues), then the customer must be able to reproduce the issue using an environment that conforms to the minimum performance requirements as defined herein.

Potential known issues include (but are not limited to) the following:

• actual usable hardware performance on virtual environments (e.g VMWare configuration and licenses)
• network overhead on remote servers (e.g. bandwidth, proxy, VPN issues, VMWare inter OS network limitations without a proper license, etc.)
• shared resources with competing applications on the same OS, or between OS on a virtual environment,
• licensing limitations (e.g. most database server licenses limit the number of usable core/CPU)
• vendor software known limitations and requirements (e.g. Oracle on VMWare vs Oracle VM)

4.2 Web Client requirements

Users only need an internet browser:

• Google Chrome v67 or newer
• Microsoft Edge v41 or newer
• Mozilla Firefox v61 or newer
• Apple Safari 11.1 or newer

4.3 Application Server Requirements

Hardware Minimum Requirements (based on physical hardware performance, not a virtual environment):

• 2 GHZ or higher quad core processor (4 cores)
• 8 GB RAM
• 10 GB of disk space (all storage is primarily in the database server, although the search index and metadata cache is on disk)

However, the Hardware Recommended Requirements for a full enterprise application server suggests at least 64 cores and 300 GB of memory, as can be calculated below:

Operating System Requirements:

• Microsoft supported Windows 64 bit versions (including Windows 2008 Server, Windows 2012 Server, Windows 2016 Server, Windows 2019 Server, Windows 7, Windows 8.x, and Windows 10).
  • Ensure that installer is executed with full Administrator privilege
  • Ensure that Microsoft .NET Framework 3.5 or higher is installed
  • Ensure that all current Microsoft Windows critical updates have been applied
• Most popular Linux/Unix 64 bit Operation System Versions (such as Redhat).
  • About Java Dependency:
    This software is not based on code compiled for any particular Operation System OS (e.g any particular Linux distribution or version ), therefore, the supported OS are 100% based on the supported OS of the underlying Java Runtime Environment (JRE) and Tomcat software (see their supported platforms for further details).
  • About Headless Linux:
    Recent versions Java Runtime Environment (JRE) such as 11 no longer bundle any fonts, and therefore rely on the underlying operating system fonts. Most bare Linux configurations bundle a minimum set of fonts, including all bare Linux VM offered on most cloud companies. However, when using an extremely bare minimum true headless Linux configuration, fontconfig and libfontconfig1 must be installed and configured on the system. Otherwise the JRE cannot access the fonts needed in order to perform diagram rendering.

Application Server Engine Requirements:

• Apache Tomcat 9 - 64 bit (bundled)
• Other Application Servers (such as IBM WebSphere or Oracle WebLogic) require manual install/setup, and are therefore not supported by this version.

Java Runtime Environment (JRE):

• OpenJDK 11 - 64 bit (bundled and recommended)
• Other Java Runtime Environment (JRE) (such as IBM Java) require manual install/setup, and are therefore not supported by this version.
  Note that Java bridges are compiled backward compatible to JRE 8

4.4 Database Server Requirements

Hardware Minimum Requirements (based on physical hardware performance, not a virtual environment):

• 2 GHZ or higher quad core processor
• 8 GB RAM.
• 20 GB of disk space (or more as needed for the data)

For small deployments (or quick proof of concepts), the EMM software package bundles PostgreSQL (for Windows only) as the EMM Database Server (which can run on the same machine as the EMM Application Server) See Application Server Installation and Configuration setup for details.

The EMM Database Server can reuse your existing Oracle, SQL Server, or PostgreSQL server:

• Oracle 12c, 18c and 19c - 64 bit

  Warning: Oracle Real Application Clusters (RAC) is not supported, especially when configured for automatic load balancing between nodes where a write on one node will need to be replicated on the other nodes.

  • The character set of the database must be AL32UTF8 (UTF8).
  • In order to find out what exact Oracle edition/version is actually installed:
sqlplus.exe SYS@<DB-NAME> as SYSDBA
   select banner from v$version where BANNER like '%Edition%';
  • In order to find out how much memory is actually available to the Oracle database, it is important to first understand how Oracle's memory is configuration and used:
    • The actual available System Global Area (SGA) memory can be found using:
sqlplus.exe SYS@<DB-NAME> as SYSDBA
   show sga;
   select * from v$sga;
   select * from v$sgainfo;
    • The actual available Program Global Area (PGA) memory can be found using:
sqlplus.exe SYS@<DB-NAME> as SYSDBA
   select * from v$pgastat;
  • In order to find out how much processing CPU/Cores is actually available to the Oracle database, query the table v$parameter for the value of cpu_count, or query the table v$license as follows:
sqlplus.exe SYS@<DB-NAME> as SYSDBA
   select * from v$license;
• Microsoft SQL Server 2008 R2 to 2022 - 64 bit

  Warning: With respect to cloud versions of SQL Server, the Microsoft Azure SQL Database and SQL Server on Virtual Machines may work if configured properly per further requirements in SQL Server setup below.
  However, RDS for SQL Server cannot not be used because you cannot grant UNSAFE ASSEMBLY (according to the documentation) to deploy the stored procedure assembly per further requirements in SQL Server setup below.

  • Detailed requirements are defined as part of the SQL Server setup below.
  • Make sure you apply the current Microsoft patches.
• PostgreSQL 13.3 (or newer)- 64 bit
  • libxml is needed, you might need to rebuild PostgreSQL with it.

Database Administrator privileges are required to install/setup/uninstall the database.

In general, one must ALWAYS install the latest service packs for a given database version BEFORE creating the EMM database. E.g., for Oracle 11.2 one is required to apply the patches to upgrade to 11.2.0.3, or whatever is the latest patch level at the time. In addition, Oracle 11.2.0.4 must have patch 17501296 applied.

Virtual Memory: For a Windows based database server, be sure to either:

• set the page file size to be managed automatically by OS
• or it should be at least 3 times the memory or RAM size for the machine.

Thus, you must have more than that much free disk space (at least 3 time the amount of memory or RAM) on the drive where the page file is defined to reside.

5. Database Server Setup

The EMM Application Server requires the connection to an existing Database server for metadata storage (metadata repository)

However, a quick install for tests or QA purpose can be achieved by using the bundle PostgreSQL database.
See the section Metadata Management (EMM) Application Server Setup for more details.

The following database setup scripts and instructions assume the following by default:
   Database Name: MM
   Database User: MM
   Database Password: = MM123! The database name and user name can be changed, and the password should of course be different.

After the product is fully installed and web connectivity has been made, one may connect to a different database by way of the web based user interface at Tools -> Administration -> Database.

5.1 Database on Oracle

Create a user MM and a database MM with the following privileges:
sqlplus.exe SYS@<DB-NAME> as SYSDBA
 
   -- Delete previous user and database if needed
   -- DROP USER MM CASCADE;
 
   CREATE USER MM IDENTIFIED BY MM123!;
 
   GRANT CONNECT TO MM;
 
   GRANT CREATE TABLE TO MM;
   GRANT CREATE VIEW TO MM;
   GRANT CREATE SEQUENCE TO MM;
   GRANT CREATE TRIGGER TO MM;
   GRANT CREATE PROCEDURE TO MM;
   GRANT CREATE TYPE TO MM;
 
   GRANT EXECUTE ON DBMS_LOB TO MM;
 
   -- If you get the error "Database exception occurred: ORA-01950: no privileges on tablespace 'USERS'"
   -- ALTER USER MM QUOTA UNLIMITED ON USERS;

5.2 Database on Microsoft SQL Server

5.2.1 Database Requirement 1 - Case Insensitive

The database must be configured to interpret SQL in a case insensitive manner.

The case insensitive collation must be Latin1_General_CI_AS.

5.2.2 Database Requirement 2 - Mixed-Authentication Mode

The Mixed-Authentication Mode is usually set during the SQL Server installation process.

The Mixed-Authentication Mode can be verified or changed by using the SQL Server Management Studio: first sign in, then right click on the root of the tree (instance of SQL Server Express), go to Security, and finally select "SQL Server and Windows Authentication mode"

5.2.3 Database Requirement 3 - TCP/IP Protocol Enabled

The TCP/IP Protocol must be enabled in the SQL Server Configuration Manager for both the named instance and the client protocols (Make sure you restart the service after changing).

5.2.4 Database Requirement 4 - Login must be Database Owner

The database login (e.g., MM) that will connect EMM to the SQL Server database must be the owner of the database.

5.2.5 Database Requirement 5 - SQL Common Language Runtime (CLR) Strict Security for SQL Server 2017 or newer

All database intensive operations (not just database maintenance, but also trace lineage, delete models, etc.) are implemented in SQL Server by stored procedures written in C# compiled/delivered in a stored procedure assembly called MIRRepo which has been signed that will be created with permission set SAFE.

If you are using SQL server 2017 or newer, then CLR Strict Security is enabled by default. Therefore, the certificate used to sign the stored procedure assembly must be imported in the database and granted the UNSAFE assembly permission (See Microsoft SQL Docs on CLR strict security) using the following commands:
   CREATE CERTIFICATE MIRRepoCert FROM BINARY = 0x308203663082024ea00302010202045eece216300d06092a864886f70d01010b05003075310b30090603550406130255533113301106035504080c0a43616c69666f726e69613116301406035504070c0d4d6f756e7461696e2056696577312a3028060355040a0c214d65746120496e746567726174696f6e20546563686e6f6c6f67792c20496e632e310d300b06035504030c044d6d4462301e170d3230303630313037303030305a170d3330303630313037303030305a3075310b30090603550406130255533113301106035504080c0a43616c69666f726e69613116301406035504070c0d4d6f756e7461696e2056696577312a3028060355040a0c214d65746120496e746567726174696f6e20546563686e6f6c6f67792c20496e632e310d300b06035504030c044d6d446230820122300d06092a864886f70d01010105000382010f003082010a0282010100c2ccf729a28a90958f71a68f6acca9f20b5c256b7c76565b2ece0cd1789bec85e9ab538ac38dc268e48c10e17d3eca1aeb14034bc67bafc05475ed013495aada683c74885f12a8bdbf2025ec3c5a0172010e7055ab27a853e77611ee6ae846453702d18ae3080977ddaee50a282b9dab3f077fe1630804b24f05c58280621dc1426fff7115e8a791435687096c09f754608bb9a6ce00002f7131f09cffd417678bddb8f7a703e4e688f2f0af501c52ecef2cbea3d37c45da4239ddb53295adaddb11dc0118b3188adf812c983d5676c5b7356d68e2258ea32cd3216db21dae49df16d2aa1aef39c618e393ce7e1b131b241c557414424fb6c17c825022a5a4270203010001300d06092a864886f70d01010b05000382010100a1db34a6cda0729a796e5ed0fe5b2f4813ff74bf96300c9ca30fb84be44bd7d0bc46c96a0726eae5e829985429ff4ff09b50ece907c5b8c7f8a71f7a16781103d7eaf2e1c7afa39e4774293610e0d04e6b0c76dc9a85891e6f5fed09059960dc7e2a7c1dc14d64aab9718747752d394b22e339da2c7e6ced1626dde991818cbcaf049d8f112a98b2aa2e80d1168f797a6c992e304e4572b4edcf40d270a281f82d7bde64e8d8b5d83574ecf5470f3d1a9d710498e133e9309a043f63b1682972678fba2a33267999795b5d040524e2f875b667dcec08d310e27b6086b2667dde70d4401fe501944f70581e559d5f3f5b72e49ff722e58594b84a8d15d5dd1414;
   CREATE LOGIN MIRRepoCertLogin FROM CERTIFICATE MIRRepoCert;
   GRANT UNSAFE ASSEMBLY TO MIRRepoCertLogin;

Alternatively, you can disable the SQL Server CLR Strict Security as follows:
   EXEC sp_configure 'show advanced options', 1;
   RECONFIGURE;
   EXEC sp_configure 'clr strict security', 0;
   RECONFIGURE;
   EXEC sp_configure 'show advanced options', 0;
   RECONFIGURE;

5.2.6 Database Preparation

Login to SQL server as a user with server admin role and execute the following commands to create a database "MM" and a user "MM" with password "MM123!" (or another one):

Enable clr, and create the database and user:
   EXEC sp_configure 'clr enabled', 1;
   RECONFIGURE;
   Go
 
   CREATE LOGIN MM WITH PASSWORD = 'MM123!';
   CREATE DATABASE MM;
   ALTER DATABASE MM SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
   ALTER DATABASE MM SET READ_COMMITTED_SNAPSHOT ON;
   ALTER DATABASE MM SET MULTI_USER WITH ROLLBACK IMMEDIATE;
   ALTER AUTHORIZATION ON DATABASE::MM to MM;

Warning: The product relies on one assembly (named MIRRepo) which is loaded from binary and not from file. This binary is created with the SAFE permissions. So in addition to being the database owner, the MM user should be granted the CREATE ASSEMBLY permission.

5.2.7 Database Connection

Advanced SQL Server Administrators may define ("hard-code") a set of TCP/IP ports for SQL Server to run over the network. However, Microsoft now recommends to run the "SQL Server Browser" service which can be done either in the Services panel or the SQL Server Configuration Manager.

For more information, read: How to: Configure Express to accept remote connections

The connection string syntax is:
   jdbc:sqlserver://<dbServer>:<dbPortNumber>;databasename=<dbName>

To connect to a named SQL server instance other than the default:

• If the SQL Server browser service is running:
  • If the named instance is configured to listen on dynamic ports:
    In the installer, specify only the instance name (in the format HOSTNAME\INSTANCENAME) and no port (the port field should be left empty), such as:
   jdbc:sqlserver://localhost\sqlexpress;databaseName=MM;
  • If the named instance is configured to listen on static IP ports:
    The SQL Server instance must be configured to run on a static TCP/IP port and that port must be specified in the installer, such as:
   jdbc:sqlserver://localhost\sqlexpress:1433;databaseName=MM;
• If the SQL Server browser service is not running:
  In the installer, specify only the instance port, such as:
   jdbc:sqlserver://localhost:1433;databaseName=MM;

To connect to SQL Server using a domain account:

1. Find the mssql JDBC driver under $MM_HOME/java/jdbc/mssql, e.g. mssql-jdbc-7.4.1.jre11.jar.
2. Download a Microsoft JDBC driver for SQL Server with the same version, e.g. 7.4, and extract the content. There you will find a sqljdbc_auth.dll from Microsoft JDBC driver x.x for SQL Server\sqljdbc_x.x\enu\auth\x64, and a mssql-jdbc-x.x.x.jre11.jar from Microsoft JDBC driver x.x for SQL Server\sqljdbc_x.x\enu
3. Copy the sqljdbc_auth.dll to $MM_HOME/bin and replace the mssql-jdbc-x.x.x.jre11.jar under $MM_HOME/java/jdbc/mssql with the one from Microsoft JDBC driver x.x for SQL Server\sqljdbc_x.x\enu
4. At the Configure Database Connection window add the string ;integratedSecurity=true at the end of the Database url, such as jdbc:sqlserver://localhost:1433;databasename=MM;integratedSecurity=true. Specify other fields and click TEST CONNECTION.

Note 1: The default database instance name for SQL Server Express is "sqlexpress, and "sqlserver" for any other SQL Server edition.

Note 2: The default SQL Server TCP/IP port number is 1433.

5.3 Database on PostgreSQL

Login to an existing database as a database superuser or a user who has CREATEROLE and CREATEDB privileges
psql.exe -h <HOST-NAME> -W -U <USER_NAME> -p <PORT> -d <DATABASE_NAME>
 
   -- Delete previous user if needed
   -- DROP USER "MM";
 
   -- If the user cannot be dropped due to any ownership issues, you'll need to reassign those objects to another user
   -- REASSIGN OWNED BY "MM" TO <OTHER-USER-NAME>;
   -- Or drop those objects
   -- DROP OWNED BY "MM"
 
   -- Create a user MM with LOGIN privilege
   CREATE USER "MM" LOGIN PASSWORD 'MM123!';
 
   -- Create a database MM with UTF8 encoding.
   CREATE DATABASE "MM" WITH OWNER "MM" ENCODING 'UTF8';

Note: For maintenance reasons, PostgreSQL database indexes can be rebuilt as follows:

• Stop the MM Tomcat server
• run the following command under the MS-DOS command prompt assuming postgresql/bin is in the system path. This may take some time if your database is large.
reindexdb --username=MM --dbname=MM -v
• Restart the MM Tomcat server

For more information on reindexdb, refer to the PostgreSQL documentation

6. Application Server Setup

6.1 Application Server Installation and Configuration

The EMM Application Server is installed as follows:

• On Linux operating systems, use tar -xjvf to extract the software package (.tbz2 format) into the directory of your choice, by default located at:
  export MM_HOME=/erwinMetadataManagement

  Depending on your software installation directory, you may need "root" privileges.

• On Windows operating systems, use unzip to extract the software package (.zip format) in the directory of your choice, by default located at:
  set MM_HOME=C:\erwinMetadataManagement

  Depending on your software installation directory, you may need "Administrator" privileges. You should avoid using the "Program Files" directories of Windows as they have are now controlled by Windows with special access rights.

  Finally, note that this admin document assumes the Linux notation for scripts and file location such as $MM_HOME/Setup.sh. The Windows scripts and file location are matching and therefore can be inferred, in the above example: %MM_HOME%\Setup.bat.

If your are using an existing database and do not wish to customize the application server (e.g. memory allocation, Windows services), then you can skip this step and go directly to the section on Application Server Execution and Initialization

Otherwise, go to the software home directory and execute the $MM_HOME/Setup.sh utility. Note On Windows, you must "Run as administrator" the %MM_HOME%\Setup.bat uility. This setup utility will allow you to setup the configuration parameters defined below through a user friendly application. After any change on any panel (tab) below, remember to press the Configure button in order to perform the configuration changes. A dialog box will be issued to confirm success or failure (with error messages).

Alternatively, this setup utility also works at the Windows command line or Linux shell, use the -? the options:
   [{ -? | --? | /? | --help }] Asking for help
   [{ -tp | --tomcat-port }] Tomcat: set the port number
   [{ -tm | --tomcat-memory }] Tomcat: set the maximum memory in Mb
   [{ -ta | --tomcat-agent }] Tomcat: switch to metadata harvesting only
   [{ -ts | --tomcat-service }] Tomcat: install or remove tomcat as a service
   [{ -s | --ssl }] SSL: enable or disable ssl
   [{ -sc | --ssl-cert-file }] SSL: the X.509 file containing the SSL certificate
   [{ -sk | --ssl-key-file }] SSL: the X.509 file containing the SSL key
   [{ -sp | --ssl-key-password }] SSL: the password for the SSL key file
   [{ -sr | --ssl-root-chain-file }] SSL: the X.509 file containing the SSL certificate chain
   [{ -ds | --database-service }] Database: install or remove database as a service
   [{ -dp | --database-port }] Database: set the port number
   [{ -dc | --database-connection }] Database: set the connection string
   [{ -du | --database-user }] Database: set the connection user
   [{ -dw | --database-password }] Database: set the connection password
   [{ -da | --database-auto-upgrade }] Database: automatically run the upgrades on startup
   [{ -mh | --mail-host }] Email: set the mail server host name
   [{ -mp | --mail-port }] Email: set the mail server port number
   [{ -mu | --mail-user }] Email: set the mail user name
   [{ -mw | --mail-password }] Email: set the mail user password
   [{ -ms | --mail-sender }] Email: set the mail sender's address
   [{ -mx | --mail-external-url }] Email: set the external URL to use in email templates
   [{ -ma | --mail-admin }] Email: a comma separated list of emails that will be notified of startup failures
   [{ -ep | --erwin-product }] Erwin: the edition to switch to: EWP or EDG
   [{ -op | --oracle-product }] Oracle: the edition to switch to: OEMM or OMM4OBI
   [{ -ch | --certificate-host }] Certificate: the host name to retrieve the certificate for
   [{ -cp | --certificate-port }] Certificate: the port number to connect to
   [{ -we | --webapp-enable }]... WebApp: enable the specified webapp (MMDoc)
   [{ -wd | --webapp-disable }]... WebApp: disable the specified webapp (MMDoc)
Some of these MM setup values (and other ones) can also be defined in $MM_HOME/conf/mm.conf.properties.

• Database Server tab:
  This is to be used only if you wish to use the bundled PostgreSQL database.
  • Enable Windows Service
    This will create the 'MM Database Server' Windows Service, set it for automatic start, and actually start it. Note that if no database already exists in $MM_HOME/postgresql/data, then a new database will be created.
    Unchecking that box will delete the 'MM Database Server' Windows Service, which is a good idea before uninstalling the EMM software. Note that your existing database in $MM_HOME/postgresql/data will not be deleted as side effect.
  • Port Number
    This is set to 5432 which is the default of PostgreSQL, but can be changed to avoid conflicts with other servers.
• Application Server tab:
  • Enable Windows Service
    This will create the "EMM Application Server" Windows Service, set it for automatic start, and actually start it. Unchecking that box will delete the "EMM Application Server" Windows Service, which is a good idea before uninstalling the EMM software.
  • Metadata Harvesting Server Only
    This allows to setup this application server as a metadata harvesting server only, rather than a full metadata management server. This is very useful in architecture deployments where the metadata management server is:
    • deployed remotely on the cloud but needs to access metadata harvesting servers (agents) locally on premise, or
    • deployed on Linux but needs to access metadata harvesting servers (agents) on a Windows machine where DM/DI/BI client tools are Windows only (e.g. COM based SDK).
  • Metadata Harvesting Browse Path
    This controls the access to the file system for metadata harvesting. The default value is set to '*' which means any Windows drive (C: and any mounted remote drive R:) or any directory from root on Linux. It is strongly recommended to limit the access to a common shared data location, and avoid system area.
  • Data Directory
    This is the location of all data files, including log files as well as the metadata harvesting caching. The data directory is located by default in the 'data' subdirectory of the application server home directory. It is recommended to separate the program data from the program files, this allows you to provide a new location for the data in a separate area (with regular backups if possible). Note that changing to a new location will not move the existing data from the previous location. Either the new location already had the data (from a previous install), or new data will be created.
  • Max Memory
    This defines the maximum memory used by Java (JRE) on the EMM Application Server (Apache Tomcat). This is unrelated to the maximum memory used by java on bridges for Metadata Harvesting which is separately set by default with the variable M_JAVA_OPTIONS in $MM_HOME/conf/conf.properties, and can be overridden within the Miscellaneous parameter of memory intensive import bridges (e.g. JDBC).
  • Port Number
    This sets a custom start port number by default to avoid conflicts with other web application servers. Note that the EMM Application Server uses 2 consecutive ports. However, this can be set back to 80 to avoid having to specify any port number in the URL.
  • SSL
    This enables Secure Socket Layer (SSL) communication for web access (HTTPS). In order to support HTTPS, the EMM Tomcat service must be configured to work with HTTPS for encryption of passwords and other content exchanged between the web client and the EMM Application Server. In this case, you will need a certificate for the HTTPS protocol to work. Note: the EMM software does not perform any error handling for validating a certificate associated with the EMM Application Server, and most web browsers will report an error if the certificate is not provided by a valid certificate authority. Thus, your certificate should be a trusted certificate provided to you by a valid Certificate Authority.
    • Certificate file
      Mandatory - Can be a .pem (privacy enhanced electronic mail), .pfx (Windows personal information exchange) or a .jks (Java keystore)
    • Root Certificate file
      Optional (only required if the above certificate file was generated by an external company as a certificate authority)
    • Key file
      Optional (often it is the case that the certificate file also contains the key)
    • SSL Key Password
      Optional (required if the key is password protected)

6.2 Application Server Upgrade

6.2.1 Understanding the Data Locations

Most application data is obviously located inside your database server, you are responsible for regular backup of such database. Upgrading your application may also upgrade the associated database content (database schema, stored procedures, indexes and of course data). It is important to understand to understand that part of the EMM software is implemented as database stored procedures like tracing the lineage. Therefore a given version of EMM corresponds to a version of the MM software in the application server (tomcat) and a version of the MM software in the database (e.g. tomcat) Therefore make sure you always backup your database before any upgrade.

Furthermore, the upgrade process may take several hours (on large repositories) and also need extra space for temp data during the migration. Therefore make sure the database has at least 20% free space.

Finally, it is also important to understand that the software installation directory (known as $MM_HOME in this document) also contains some critical application data and application setup customizations that have to be taken into account in your backup or upgrade process, including:

• postgresql/data
  which contains the actual PostgreSQL database data only on Windows and only when configured with the Setup utility (in the "Database Server" tab).
• data
  which contains other application data, including:
  • download/
    • MIMB/ contains third party download software packages
  • files/
    • MIMB/ contains files for upgrade of MIMBWebServices
    • mm/
      • analytics/ contains the generated analytics data files (new beta feature).
      • backups/ contains model backups for model comparison purpose.
      • operations/ contains the operation execution generated files such as a troubleshooting package or a backup ready to download. As these generated files can be very large, it is critical to execute the operation "Delete operation logs and files" either manually (MANAGE > Repository, right click on at the repository root) or automatically (MANAGE > Schedules).
      • sessions/ contains any temporary files for user login session.
      • tmp/ contains other temporary files.
  • logs/ for the log files tomcat and mimb
    • MIMB/ contains the model import/export bridge log files.
    • search/ contains the search query log files.
    • tomcat/ contains the tomcat log files
  • MIMB/
    • cache/ contains bridge execution files organized as BridgeId/ImportId/mir_Version,nativeVersion
    • parameters/ contains bridge execution parameter files (only for some bridges like DI/ETL with runtime variables) organized as BridgeId/ImportId/Parameters
  • search/ contains the lucene search engine indexes (they will be automatically fully rebuilt from scratch if the folder is empty which will take a lot of time)
  • temp/ for any temporarily files (including from the mimb bridges)
  • webapps/ for application server (tomcat) cache
  Remember that the actual location of this EMM Application Server data directory can be configured with the Setup utility (in the "Application Server" tab).
• conf which contains the MM configuration / customizations organized as follows:
  • conf.properties contains file containing most customizations defined with the Setup utility (in the "Application Server" tab)
  • ModelBridgeList.xml contains the list of enabled bridges and their names
  • resources/ directory containing any User Interface Customizations, in particular MM.properties and MetadataExplorer.xml.
  • Template/ contains the default template files of all the above files/directories, including a potentially new updated ModelBridgeList.xml or MetadataExplorer.xml after cumulative patches.
• tomcat/conf
  with the tomcat.properties file containing the tomcat port and memory customizations defined the Setup utility in the "Application Server" tab, and the keystore file containing the tomcat SSL certificates defined with the Setup utility (in the "Application Server" tab).
• jre/lib/security
  which also contains some SSL customizations defined with the Setup utility (in the "Application Server" tab). It is recommended to not reuse such directory, but rather reinstall the SSL keys with the Setup utility.

6.2.2 Upgrade Process

We recommend the following upgrade process:

1. Stop your EMM application server in the same way it was started such as stopping the Windows services / Linux daemon, stopping the desktop command windows, or using the tomcat/bin/shutdown. Remove the application server service if one was created (Windows only).
2. Stop your EMM database server and ONLY if you are using the bundled PostgreSQL database on Windows then use the $MM_HOME/Setup.bat to remove the Database Service as follows: Go to the Database Server tab, enable disable or uncheck the database service check box, and click on the Configure button.
3. Backup your EMM data including your database and data file directories as explained above.
4. Backup your EMM software by copying the $MM_HOME directory as $MM_HOME_OLD.
5. Install the complete EMM new software (ONLY needed for clean install of a new version) by deleting the old $MM_HOME, and then creating a new one by unzipping the new EMM full package.
6. Apply the latest EMM software cumulative patch by unzipping it from inside the new $MM_HOME directory.
   • WARNING 1: Make sure you unzip with overwrite on Windows, and use unzip -u on linux to update files while retaining permissions.
   • WARNING 2: You cannot use cumulative patches for major version upgrades, you must first start from a clean install of the new major GA version.
   • WARNING 3: You cannot reverse / unzip an older cumulative patch, you must restart from a clean install of the original GA version.
7. Restore your EMM data and software customization/setup (ONLY needed for a clean install of a new version) by copying the appropriate files and directories (as previously explained) from $MM_HOME_OLD to $MM_HOME, including at least $MM_HOME/data and $MM_HOME/conf/conf.properties but possibly more as used and customized such as $MM_HOME/conf/resources, or $MM_HOME/tomcat/conf.
8. Integrate the EMM new software features in your configuration by copying potential new versions of files from $MM_HOME/conf/Template into their matching directories in $MM_HOME/conf/. For example the new $MM_HOME/conf/ModelBridgeList.xml may contain some new or updated bridges. WARNING: if you had customized some files such as $MM_HOME\$MM_HOME/conf/resources/MM.properties, you must re-apply/merge such customization starting from the new version of that file copied Template.
9. Reconfigure your EMM Database Server (ONLY if you are using the bundled PostgreSQL database on Windows). As the PostgresSQL server software version may have changed and the database server needs to be upgraded. Therefore, you must:
   1. Execute the (old renamed installation) $MM_HOME_OLD/Setup.sh utility to restore the old Database Service as follows: Go to the Database Server tab, enable the database service check box, and click on the Configure button.
   2. Execute the (new installation) $MM_HOME/Setup.sh utility as follows: go to the Database Server tab, enable the database service check box, and click on the Configure button. At this point the $MM_HOME/Setup.sh utility will retrieve the existing PostgreSQL data from the old install directory (which was already running as a service), will migrate them to the new install directory, and will remove the old PostgreSQL service, before starting the new PostgreSQL service on the new directory.
10. Restart your EMM database server (ONLY if you are using the bundled PostgreSQL database on Windows).
11. Restart your EMM application server after which your first login as Administrator may prompt you for an upgrade of your MM database.
12. Redo all above steps for any other EMM Application server configured as MIMB harvesting agent.
13. Update your EMM repository content (ONLY as needed) if the upgrade contains new and improved import bridges that would require to fully re-import the model (remove incremental harvesting in such case) and this will therefore require to re-build the related Configurations.

Typical DevOps Linux bash scripts include the following commands:


#! /bin/bash

##############
# MM INSTALL #
##############

######## START SETUP ########

sudo su

MM_INSTALL=/opt
MM_INSTALLER=EMM-12.0-20221231
MM_SOFTWARE=ErwinMetadataManagement
MM_HOME=${MM_INSTALL}/${MM_SOFTWARE}

######## DOWNLOAD INSTALLER TO INSTALL DIRECTORY ########

# get $MM_INSTALL/$MM_INSTALLER

######## STOP SERVER ########

$MM_HOME/ShutdownServerApplication.sh

# double check for any tomcat process still running
# ps -edf | grep tomcat
# kill -9

######## FULL CLEAN INSTALL NEW SERVER ########

# Backup old install and extract new one
cd $MM_INSTALL
mv $MM_SOFTWARE $MM_SOFTWARE.yyyymmdd
tar -xvf $MM_INSTALLER.tbz2
rm MM_INSTALLER.tbz2

# optional ThirdParty software extract for brand new install on machines with no access to internet
mv *ThirdParty*.zip $MM_SOFTWARE
cd $MM_SOFTWARE
unzip *ThirdParty*.zip
rm *ThirdParty*.zip

######## COPY SETUP AND DATA ########

cd $MM_INSTALL_DIR/$MM_SOFTWARE.yyyymmdd

# data directory with large MM search index and MIMB cache
# cp -r data ../$MM_SOFTWARE
mv data ../$MM_SOFTWARE

# conf properties (memory, path, etc.)
#M_BROWSE_PATH=
#M_JAVA_OPTIONS=-Xmx64G
cp conf/conf.properties ../$MM_SOFTWARE/conf

# tomcat conf properties (URL, ports, services, etc)
cp tomcat/conf/tomcat.properties ../$MM_SOFTWARE/tomcat/conf

######## HARVESTING AGENT SERVERS ########

# conf agent identification to MM server
cp conf/agent.properties ../$MM_SOFTWARE/conf

# conf agent access to SSL based MM server
$MM_HOME/Setup.sh -ch abc.def.com -cp 443

######## MM MAIN SERVER ########

# MM conf properties (database connection, email setup, etc.)
cp conf/mm.conf.properties ../$MM_SOFTWARE/conf

# MM resources such as the UI look & feel
alias cp=cp
cp -rf conf/resources ../$MM_SOFTWARE/conf

# Tomcat SSH Setup
cp tomcat/conf/keystore ../$MM_SOFTWARE/tomcat/conf
cp tomcat/conf/server.xml ../$MM_SOFTWARE/tomcat/conf

######## RESTART SERVER ########

$MM_HOME/RestartServerApplication.sh

######## END ########

 

6.2.3 Version Specific Upgrade Issues and Recommendations

Upgrading to a new version may have version specific issues or recommendations that are listed at the bottom of the release notes: see Release Changes for more details.

6.2.4 Upgrade and Migration Best Practice

The following critical steps represent the best practice in EMM Server upgrade or migration to a new machine (on prem or cloud)

Backup

As with any migration / upgrade process, it is critical to backup the underlying data:

1. the installation data and conf directories (see Understanding the Data Locations)
2. the repository database. (see Database Server Backup/Restore)

Repository Cleanup

One of the most critical first step is to save disk space and speed up performance by performing a major cleanup of the repository:

1. Apply the latest MIMM and MIMB Cumulative patches (in the EMM Server and all MIMB Harvesting Agents).
2. Make sure that all Configurations are NOT on auto update.
3. Stop any scheduled operations, including all automatic metadata harvesting and database maintenance (MANAGE > Schedules).
4. Delete all unused / test Directories, Configurations, Models, Mappings, etc. (MANAGE > Repository).
5. Delete as many versions as possible (e.g. retaining the last few versions) of the remaining critical Configurations (MANAGE > Repository).
6. Delete unused model versions (MANAGE > Repository: Repository object > Operations > Delete unused versions).
7. Run the database maintenance to purge all deleted objects from the database (MANAGE > Schedules: Run Database Maintenance). Note that each database maintenance run purges deleted models for only 2 hours, therefore the database maintenance has to run as many times as needed until the log no long shows any models to delete.

Model Imports

In order to avoid surprises after the upgrade/migration, it is critical to make sure that you started with a stable environment. The most important aspect of that is to make sure all imports are working before any upgrade because the source may no longer be available, may not have been imported with the latest version of the bridge, or may simply not have been imported for a long time.

1. Apply the latest MIMM and MIMB Cumulative patches (in the EMM Server and all MIMB Harvesting Agents).
2. Make sure that all Configurations are NOT on auto update.
3. Stop any scheduled operations, including all automatic metadata harvesting and database maintenance (MANAGE > Schedules).
4. Delete the model import cache in $MM_HOME/data/MIMB/cache).
5. Manual full import (no incremental harvesting) of all Models (one by one) with clear import cache and uncheck "Create new versions only when new import has changes".
6. Manual force build of all Configurations (one by one) with testing (connection stitching as needed for lineage).

Repository Database

If the EMM repository database server has to be migrated to a new machine (e.g. from on prem to cloud), make sure you follow the proper the proper database vendor process (see Database Server Backup/Restore).

Note that the above process works well as long as you stay with the same database technology (e.g. PostgreSQL), but such database cannot be performed between database technologies, such as Oracle or SQL Server to PostgreSQL because the repository database implementations are different / optimized for each database technology. In such case, one can use the EMM Application's Backup/Restore format (directories of XML files) but there are known limitations as some content is not back up in that format.

Harvesting Agents

If the EMM Application Server has to be migrated to a new machine (e.g. from on prem to cloud), Make sure to reconnect each MIMB harvesting agents to the new EMM Application Server.

Note that such MIMB harvesting agents can remain on prem, while connecting to the new EMM Application Server on cloud.

6.3 Application Server Execution and Initialization

The easiest way to start the EMM Application Server is to execute the $MM_HOME/RestartApplicationServer.sh utility. On Windows, you must "Run as administrator" the %MM_HOME%\RestartApplicationServer.bat utility.

• On Windows operating systems, you can alternatively use the Windows Services to control the EMM Application Server by using the %MM_HOME%\RestartApplicationService.bat utility instead. This utility will create the Windows Service for the EMM Application Server, if it was not already created by previous execution of this utility or the Setup.bat utility. At this point, you can simply use the Windows Services to start, stop or restart the EMM Application Server automatically.
  When running the EMM Application Server as a Windows Service, it is important to configure the user running such service in order to have full access rights to the needed files and applications. For example, the MIMB bridges involved in the metadata harvesting may need to invoke the SDK of third party software such as the COM based API of ERwin, or SAP BusinessObjects Universe Designer.
  In order to set such access rights, go to the services manager of Windows, right-click on the EMM Application Server service. Then, go to the "Log On" tab to define an account by name under which the service will run.
• On Linux operating systems, administrators can use the system daemon directories (e.g. /etc/init.d/ or /etc/systemd/) to control the EMM Application Server (either using the RestartApplicationServer.sh utility or directly controlling the tomcat server in the home directory).

The final initialization steps of the setup are performed over the web browser as follows:

1. Connection
   Connecting to the server on Windows can be simply achieved by opening the Metadata Management link in the home directory. In all cases, you can connect to the server using your internet browser to open by default: http://localhost:11680/MM. Note that the default port of this URL number may have been changed by the Setup utility in the section Server Installation and Configuration..
2. Database
   Define the connection to the previously created database (in the above steps), by providing the database type, user, password, and URL (JDBC connection). If you are using the PostgreSQL database bundled with the software package for Windows, then all these parameters should be already preset. Press Test Connection button to verify proper database connectivity. Finally, when the pressing the Save button, the EMM Application Server will create all the necessary tables in the database.
3. License
   Click on the Host Details link to obtain the obtained your HostInfo.xml file that should be sent with your license request. Warning: Make sure your are NOT connected to any VPN during that step, then your license will work independently of your VPN connection. After you have received your MM.lic license file, browse for it and click on the Save License button.
   Alternatively, the host info file can be generated at the Windows command line using $MM_HOME/bin/HostFileGen.bat, or Linux shell using bin/HostFileGen.sh. This will generate your HostInfo.xml file (in the bin directory) that should be sent with your license request. After you have received your MM.lic license file, copy it to $MM_HOME/conf/MM.lic and restart the application server.
4. Login
   Login as "Administrator" with password "Administrator". Note that you should change that password later in the application by going to: Tools -> Administration -> Users)

6.4 Custom integration with authentication environments

EMM is able to support three authentication methods:

1. Native Authentication, where the password is managed by the software and stored within the database.
2. LDAP Authentication, where the software does not manage or store the LDAP passwords at all. Instead, it is simply passed it through to LDAP in order to authenticate.
3. External Authentication such as Single Sign On (SSO), where the software does not perform any authentication, and leaves that responsibility to a local single sign on service managed by the customer.

In Tools->Administration->Users one may specify either:

1. Mixed Native and LDAP authentication where users may be authenticated either as native users or LDAP users
2. External authentication where the system does not perform any authentication, leaving it up to a local Single Sign On environment.

6.4.1 Native Authentication Configuration Issues

There are no specific configuration steps for Native Authentication.

6.4.2 LDAP Authentication Configuration Issues

There are no special server configuration issues for LDAP Authentication. LDAP connectivity configuration is documented in the online help.

6.4.3 Windows Authentication Issues

It is also possible to enable the Application Server to obtains authentication for users from Windows authentication via the browser (client). This way, users will automatically be authenticated if they are running from a Windows session.

To do so, one must install a third party product named Waffle (Windows Authentication Functional Framework) as an addon (see here):

1. Please ensure that all LDAP settings are correct and users are able to log into the product via LDAP authentication. LDAP connectivity configuration is documented in the online help.
2. Unzip the Waffle zip.
3. Copy all the jar files from it to $MM_HOME/tomcat/lib
4. Open $MM_HOME/tomcat/conf/web.xml. Search for "Windows authentication support". Uncomment the block following that.
5. Restart EMM.
6. You should have windows authentication enabled now. Any valid windows user will be logged in as guest by default as long as licensing allows it. If you need to get an administrator interface, you can access: http://host:port/MM/Auth?nativeLogin (optionally you can force a redirect to either &redirectTo=/MM/Explorer or /MM/Manager)
7. Provide connection information for the database you created above.

Note: Waffle is designed around Windows libraries and thus it is recommended that you use a Windows OS based machine as the Application Server. While it is possible to use Waffle on a Linux based machine, it will require a great deal of manual setup and compilation. Please follow the Waffle documentation for such an implementation (see here).

Note: When using Waffle on a Windows based Application Server (as is recommended) you must run as run the EMM software as a Windows service (not as an Application) in order for Waffle to work properly.

Note: Automatic Windows authentication will not allow one to use the browser refresh (f5) with IE 8.x when used as the client browser. Refresh will force a re-authentication on IE 8.x browsers and will not be automatically authenticated. If this occurs, the user must close all instances of the browser and start again. To avoid this issue, one must use IE 9.x or later or another approved browser (see System requirements) In addition, for Firefox, you must configure the browser at each client to support automatic Windows authentication. Please refer to the Waffle web site here.

6.5 Custom integration for Secure Socket Layer (SSL) communication

6.5.1 Configuring SSL for HTTPS access from Web Clients

The recommended method of configuring HTTPS access to Web Clients is using the -ssl options of the Setup utility as explained in Server Installation and Configuration.
   [{ -s | --ssl }] SSL: enable or disable ssl
   [{ -sc | --ssl-cert-file }] SSL: the X.509 file containing the SSL certificate
   [{ -sk | --ssl-key-file }] SSL: the X.509 file containing the SSL key
   [{ -sp | --ssl-key-password }] SSL: the password for the SSL key file
   [{ -sr | --ssl-root-chain-file }] SSL: the X.509 file containing the SSL certificate chain
For example:
   $MM_HOME/Setup.sh -s true -sk MyPrivateKeyFile -sp MyPrivateKeyPassword

If the above method fails, you may manually update the server keystore location and password, by editing $MM_HOME/tomcat/conf/server.xml to change the value of certificateKeystoreFile and certificateKeystorePassword within the <Certificate> section.

6.5.2 Configuring SSL to access Remote Servers

There are multiple use cases of using SSL to access remote server:

• Configuring HTTPS for Remote Metadata Harvesting Agents (Remote MIMM Server)
• Configuring SSL for Harvesting Metadata with API based bridges (e.g. a database server via JDBC)
• Configuring SSL for Repository Storage (database server via JDBC)
• Configuring LDAPS for Enterprise Directory

In all above uses cases, no extra setup is needed when using an official certificate signed by a Certificate Authority (CA), as using the https based URL is enough.

However, when using a self signed certificate, then such a certificate needs to be explicitly imported in the java environment of your server. The recommended method to import such a certificate is using the -certificate options of the Setup utility as explained in Server Installation and Configuration.
   [{ -ch | --certificate-host }] Certificate: the host name to retrieve the certificate for
   [{ -cp | --certificate-port }] Certificate: the port number to connect to
For example:
   $MM_HOME/Setup.sh -ch MyServer.MyDomain.com -cp 443
If the above method fails, you may manually import the certificate into the java environment keystore as follows:
   cd $MM_HOME/jre/lib/security
   mv jssecacerts jssecacerts.old
   $MM_HOME/bin/keytool -importkeystore -srckeystore YourSelfSignedCertificate -keystore jssecacerts
   $MM_HOME/RestartServerApplication.sh (or RestartServerService.bat on Windows)
Note that the above import steps have to be repeated for the self signed certificate of every remote servers.

6.6 Security and Vulnerability Considerations

For extra security and vulnerability reasons, the Apache Tomcat bundled within EMM can be updated as follows:

6.6.1 Tomcat Upgrade To The Current Patches

Any new version EMM includes the latest version of Tomcat for bug fixes, performance, security and vulnerability reasons.
According to the tomcat 9 changelog https://tomcat.apache.org/tomcat-9.0-doc/changelog.html, the last Common Vulnerabilities and Exposures (CVE) fix was made in 9.0.30. Therefore, there is no real benefit to upgrade to the latest version.
In addition, it is not recommended that you manually install / patch the version of Tomcat bundled in EMM, as Tomcat new versions may not be compatible and require more files to be copies. The following process describe the patch process anyway although not officially supported:

1. Check the version of Apache Tomcat that was bundled within your EMM (e.g. Tomcat 9.0.45) which is expressed in:
     $MM_HOME/Documentation/License/MIMM-ThirdParty-LICENSES.html and
     $MM_HOME/tomcat/RELEASE-NOTES
2. Download the latest patch version (e.g. 9.0.46) within the same major version:
    https://tomcat.apache.org/download-90.cgi
3. Unzip in temporary directory
4. Stop Tomcat
5. Copy the lib directory from that temporary directory to the $MM_HOME/tomcat/lib (overwriting files if necessary)
6. Restart Tomcat

6.6.2 Tomcat Check For Allowed Referrer

For additional protection, we recommend enabling Tomcat to check for allowed referrer:

1. Edit $MM_HOME/tomcat/conf/web.xml
   1. Uncomment the two filter sections in the 'Checks referrer is allowed' section. The variable ${server.fqdn} will be substituted with the value of M_SERVER_FQDN in tomcat.properties
2. Edit $MM_HOME/tomcat/conf/tomcat.properties by:
   1. Changing the M_SERVER_FQDN variable from localhost to <myMMServer.myDomain>

6.6.3 Tomcat Configuration to avoid TLS violation of Cryptographic Standard STD-IT-0005

EMM uses Java 11 that supports Transport Layer Security (TLS) 1.3, but it is compiled for Java 8 that supports TLS 1.2. To ensure that Tomcat does not "drop down to using" a lower version (1.1. and 1.0), one must configure Tomcat appropriately.

For more details on configuring Tomcat SSL, see Tomcat's documentation at:
https://tomcat.apache.org/tomcat-9.0-doc/config/http.html#SSL_Support

6.6.4 Tomcat Configuration to include HTTP security headers

For additional protection, you can edit $MM_HOME/tomcat/conf/web.xml

By default, the application sets the following to the recommended values:
   * Content-Security-Policy
   * X-Content-Type-Options
   * X-XSS-Protection

The X-Frame-Options is not set by default, it can be done manually by adding the following fragment:
   <init-param>
      <param-name>X-Frame-Options</param-name>
      <param-value>sameorigin</param-value>
   </init-param>

The HSTS headers are not necessary as when the application is configured for HTTPS then HTTP is not allowed at all, and do not provide automatic redirection. However, you may want/need to add it, you can do so manually by adding the following fragment:
   <init-param>
      <param-name>Strict-Transport-Security</param-name>
      <param-value>max-age=31536000; includeSubDomains</param-value>
   </init-param>

6.6.5 Tomcat Configuration for Access Control with Valve

The first line of defense of the Tomcat based primary EMM server or any of its metadata harvesting agents / MIMB servers requires a proper firewall setup. As a last line of defense, the tomcat remote address valve can be configured for access control.

In such case the Tomcat of any metadata harvesting agents / MIMB server should configure with a valve to only accepts requests from its associated primary MM server.
This requires to edit the file tomcat/conf/MetaIntegration/localhost/MIMBWebServices.xml
in order to add the following line:
<Valve className="org.apache.catalina.valves.RemoteAddrValve" allow="127\.\d+\.\d+\.\d+|::1|0:0:0:0:0:0:0:1"\>

For more details on configuring Tomcat access control with valve, see Tomcat's documentation at:
https://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Remote_Address_Valve

6.6.6 Secret / Password Encryption

There are a few cases where any account secret / user password is stored in the EMM repository database using an encryption method that is two-way in order to restore the original password just before calling a third part API later:

1. When configuring metadata harvesting (Model > Import > Setup), some bridge parameters require authentication to the source technology / server (e.g. user / password of a database or a BI server)
2. When configuring LDAP based authentication (MANAGE > Users > LDAP)
3. When configuring Email notification (MANAGE > Email Notification)
4. When configuring Cloud Identity (MANAGE > Cloud Identity)

Because of this requirement, EMM cannot use key-based industry standard encryption.
EMM instead stores such user/password in the repository database (i.e. at rest) using a confidential proprietary reversible encryption algorithm based upon industry standards.

NOTE 1: A second level of encryption can also be used during transport (i.e in motion) using 6.5 Custom integration for Secure Socket Layer (SSL) communication

1. HTTPS for remote metadata harvesting from the main EMM Server and a remote Harvesting Agent / Server.
   See 6.5.1 Configuring EMM to securely connect via HTTPS to another EMM server for Metadata Harvesting
2. LDAPS for authentication to the Enterprise Directory.
3. When using LDAP based authentication.
4. See 6.5 2 Configuring EMM to securely connect via LDAPS to the Enterprise Directory

NOTE 2: Alternative secret / password encryption and external storage solutions are available using Cloud Identity and Cloud Secret Vaults (such as Amazon Web Services, Microsoft Azure, or Google Cloud).
See, MANAGE > Cloud Identity

6.7 Lucene Search Engine Troubleshooting

The EMM search capabilities are implemented by a Lucene search engine with indexes located in in $MM_HOME/data
If this Lucene search index directory has been lost, the EMM server will automatically recreate it.

If the Lucene search index has been corrupted for any reason (e.g power outage during indexing, out of memory, concurrent write to the index, etc.), then the search index directory can be deleted and the EMM server will automatically recreate it.
Although not officially supported, the Administrator can attempt to use use the Lucene CheckIndex to exorcise corrupted documents from the index.
Here are the steps you can following (replace lucene-xxxxxxxx with the actual directory name of your search index).

1. cp -R $MM_HOME/data/search/lucene_xxxxxxxx /backup
2. cd /tmp
3. mkdir CheckIndex
4. cd CheckIndex
5. $MM_HOME/jre/bin/jar -xvf $MM_HOME/tomcat/webapps/MM.war
6. cd WEB-INF
7. java -classpath "lib/*" -ea:org.apache.lucene... org.apache.lucene.index
8. CheckIndex $MM_HOME/data/search/lucene_xxxxxxxx
9. Examine the output of the above command to see if there is any corrupted segment. If there is run the same command above with an extra option "-exorcise".
10. You may delete the CheckIndex directory after you are done.

For more details, see more about about Lucene CheckIndex.

7. Metadata Harvesting Model Bridge (MIMB) Setup

The Metadata Integration or Metadata Harvesting from third party databases, data modeling, data integration or business intelligence tools is performed by the integrated Meta Integration® Model Bridge (MIMB) software. By default, the installer software deploys and configures both EMM and MIMB on the same machine, where the EMM Application Server accesses the MIMB Web Services locally. MIMB can also be installed and configured as a remote MIMB Agent on another machine. This is very useful in architecture deployments where the metadata management server is:

• deployed remotely on the cloud but needs to access metadata harvesting servers (agents) locally on premise, or
• deployed on Linux but needs to access metadata harvesting servers (agents) on a Windows machine where DM/DI/BI client tools are Windows only (e.g. COM based SDK).

Essential customizations (e.g. directories, memory) of the MIMB Application Server can be performed in the following configuration file:
   $MM_HOME/conf/conf.properties

Recommended customizations include:

• M_BROWSE_PATH to browse local and mapped network drive.

  All metadata harvesting file and directory parameter references are relative to the server. The reason is that the server must have access to these resources anytime another event (e.g., scheduled harvest) is to occur. When harvesting a model, then, the UI presents a set of paths that may be browsed in order to select these files and directories. Setting the M_BROWSE_PATH parameter allows one to define which drives and network paths will be available in the UI. One may update the M_BROWSE_PATH using the UI (on the application server) presented by the $MM_HOME/Setup.sh utility (see also Application Server Execution and Initialization), or by editing the $MM_HOME/conf/conf.properties file directly.

  On installation, the set includes all directly attached drives., which is specified by an asterisk "* as follows M_BROWSE_PATH=*.

  Note for Windows based application servers: When running as a service, the drive names (mapped) and paths may not be the same as what a user sees when logged in, and thus the "*" value will not be see all drives you might expect when selecting drives using the UI. Instead, one must explicitly list all the drives and network paths that one wants to be available to all users in the UI. Also, it is not sufficient to simply enter the mapped drive id (e.g., "N:\"), as that drive mapping is also generally not available to services. Thus, one should specify the physical drives by letters, but must specify the network paths completely, such as M_BROWSE_PATH=C:\, E:\, \\network-drive\shared\

  Note that the above also applies even to script backup and restore drives.

• M_DATA_DIRECTORY to relocate the data such as the log files, and metadata incremental harvesting cache as needed for very large DI or BI tools.
• M_JAVA_OPTIONS to increase the maximum memory used by java bridges during the metadata harvesting of very large DB, DI or BI tools. Note that this parameter defines the default maximum for all java bridges, however most memory intensive java bridges (e.g JDBC bridges) have the ability to define its own maximum memory in their last parameter called Miscellaneous.

When the MIMB Application Server is used a local metadata harvesting agent connected to a EMM Application Server on the cloud, the additional customizations are needed in the $MM_HOME/conf/agent.properties configuration file where:

• M_SERVER_URL is the URL of the EMM Application Server on the cloud such as M_SERVER_URL=http://server:11680/MM.
• M_AGENT_NAME is the agent name such as M_AGENT_NAME=MyCompanyOnPemise that the above EMM Application Server will then use to refer to this metadata harvesting server agent.

8. User Interface Look & Feel Customization

8.1 Login and Headers

Customize the following files and directories using the embedded instructions (in comments):
   $MM_HOME\$MM_HOME/conf/resources/MM.properties
   $MM_HOME/conf/resources/FavIcon.ico (can be .png)
   $MM_HOME/conf/resources/LoginLogo.svg (can be .png)
   $MM_HOME/conf/resources/HeaderLogo.svg (can be .png) with the following requirements:
       - Height must be exactly 27px (required)
       - Ideally single color (recommended)
       - Color should match miti.mimm.header.fontcolor (recommended)
       - Color must be compatible (look nice) with selected miti.mimm.header.bgcolor (recommended)
   $MM_HOME/conf/resources/web (optional advanced only)

8.2 Metadata Explorer for Business Users

Customize the following files using the embedded instructions (in comments):
   $MM_HOME\$MM_HOME/conf/resources/MetadataExplorer.xml

9. REST API SDK

The REST API SDK documentation is available within the UI by going to the Help menu (top right corner) under Help on REST API, or go directly to: http://localhost:11680/MM/REST-API/.

10. Database Server Backup/Restore

The EMM Application Server requires the connection to an existing Database server for metadata storage (metadata repository). Metadata Integration recommends to backup the EMM metadata repository regularly and especially before any upgrade.

This document describes the commands and instructions to perform the EMM repository database backup and restore tasks. These database commands and instructions assume the following by default:

• Database Name: MM
• database User: MM
• Database Password: MM123!

The database name and user name can be changed, and the password should of course be different.

We assume that objects from backup are restored into the same database later.

Always use the same database software version to perform the backup and restore. Backups created by more recent version of a database server may not be restored in earlier versions of the database server.

Note that any data that is saved after a backup is done will be lost if you restore the backup.

Stop the EMM Application Server before you perform the backup and restore tasks. Restart the EMM Application Server afterwards.

To ensure the optimal EMM Application Server performance after a restore operation, run the database maintenance script in the EMM Management UI using Tools → Administration → Schedules → Run Database Maintenance to update the database statistics.

Backup and restore on a very large EMM repository database may take a long time. Refer to the database backup and restore documentation to enable parallelism, incremental backup and restore for better performance. The instructions given below are for a full database backup and restore.

10.1 Database on Oracle

We can use the Oracle Data Pump technology for Oracle 10g, 11g, 12c, 18c and 19c database.

First create an Oracle directory BACKUP_DIRECTORY that points to an operating system directory on the database server machine for reading and writing files.

Assume ORCL is the database server name or SID.
Sqlplus.exe / as sysdba
   CREATE OR REPLACE DIRECTORY BACKUP_DIRECTORY as '/Databases/Backups/Oracle';
   GRANT read, write ON DIRECTORY BACKUP_DIRECTORY TO MM;

Then use Oracle Data Pump (expdp, impdp) to backup and restore the EMM metadata repository.

Backup

To backup (export) the EMM metadata repository database to a file MM.dmp and write the export log to expdpMM.log in the operating system directory /Databases/Backups/Oracle:
   expdp MM/MM123! schemas=MM directory=BACKUP_DIRECTORY dumpfile=MM.dmp logfile=expdpMM.log

Restore

Before you restore the backup to the EMM repository database, you need to drop the schema MM to delete existing objects and data from the EMM repository database. Restore will recreate the schema MM.
   Sqlplus.exe SYS@ORCL as SYSDBA
      DROP USER MM CASCADE;

To restore (import) the EMM metadata repository database from a file MM.dmp and write the import log to impdpMM.log in the operating system directory /Databases/Backups/Oracle:
   Impdp schemas=MM directory=BACKUP_DIRECTORY dumpfile=MM.dmp logfile=impdpMM.log

When prompted for Username, enter / as sysdba.

You may refer to the Oracle Data Pump documentation for more details on the expdp and impdp commands. Backup Restore using Recovery Manager (RMAN) You may also use the Oracle Recovery Manager (RMAN) to backup and restore your EMM repository database. It is a good practise to create a separate table space for EMM repository database and restore only from that table space. For more information refer to the Oracle Database Backup and Recovery User’s Guide.

10.2 Database on SQLServer

To perform backup or restore log in to the SQL Server Management Studio, open a new query window and execute the backup or restore commands given below. You can also use the SQL Server Management Studio Object Explorer UI to perform the backup and restore tasks.

Backup

To backup the EMM repository database MM into a file /backups/SQLServer/MM.bak, use the following command:
   BACKUP DATABASE [MM] TO DISK = N'/backups/SQLServer/MM.bak' WITH NAME = N'MM-Full Database Backup'
   GO

Restore

To restore the file /backups/SQLServer/MM.bak into the EMM repository database MM use the following commands.
   USE [master]
   GO
   ALTER DATABASE MM SET SINGLE_USER WITH ROLLBACK IMMEDIATE
   GO
   RESTORE DATABASE [MM] FROM DISK = N'/backups/SQLServer/MM.bak' WITH REPLACE
   GO

You may refer to the Microsoft SQL Server backup and restore documentation for more details on the above commands.

10.3 Database on PostgreSQL

Backup

To backup the EMM repository database MM into a file /backups/PostgreSQL/MM.dmp use the following command:
   pg_dump -b -f /backups/PostgreSQL/MM.dmp -F t -d "MM" -h localhost -w -p 5432 -U MM

Restore

To restore the file /backups/PostgreSQL/MM.dmp into the EMM repository database MM use the following command. All database objects and data are dropped before recreating them.
   pg_restore -c -F t -d "MM" -h localhost -w -p 5432 -U MM --if-exists /backups/PostgreSQL/MM.dmp

You may refer to the PostgreSQL pg_dump and pg_restore documentation for more details on the above commands.