This section describes three tools. These are meant for migration scenarios of Content Management Server in multi-environment setup, as for example a development and a production stage, related to UUIDs of contents. Another application is transfer of UUIDs from a Content Management Server to its Master Live Server and Replication Live Servers. For details on UUID support in CoreMedia Content Cloud see Section 5.2, “UUIDs” in Unified API Developer Manual.

To finish the process, a server downtime is required. Thus, please plan your upgrade scenario carefully. Find more details at Section “Preparation”.

Purpose

The tool set has the following applications:

To keep UUIDs synchronized between two environments after an initial migration step, consider using serverexport and serverimport with enabled UUID export. For details see Section 3.13.2.17, “Serverimport/Serverexport”. If you forgot to include UUIDs during content transfer with serverexport and serverimport, you may repeat synchronization of UUIDs with the tools described here.

UUIDs of contents will be synchronized to Live Servers automatically upon publication. Therefore, the tools typically need to be applied only once within a single environment.

Example Scenario: Multi-Environment Migration

For the following example, this setup is expected:

Typical Usage

The following is a typical flow of actions when migrating UUIDs. Assuming that your Development Environment is hosted at dev.host and your Production Environment is hosted at prod.host the following shows the order of commands. Remember that you need to shut down your server at Production Environment before running the last command:

$ cm content-uuid-export \
    --user admin \
    --url https://dev.host/coremedia/ior \
    --query "BELOW PATH '/Sites'" \
    --dburl jdbc:mysql://dev.host:3306/cm_management \
    --dbuser cm_management \
    --dbpassword ... \
    --output dev-export.csv

$ cm generate-content-uuid-map \
    --dburl jdbc:mysql://prod.host:3306/cm_management \
    ...
    --input dev-export.csv \
    --output prod-map.csv

$ cm content-uuid-import \
    --dburl jdbc:mysql://prod.host:3306/cm_management \
    ...
    --input prod-map.csv

Note, that you not only have to shut down your server, but also restart any client. This applies to clients with the ability to reconnect as well, as they will not get notified on changed UUIDs.

Preparation

To finish the process of UUID synchronization you will require a server downtime for Production Environment when using the last of the three tools. For a rough estimation on possible downtimes see Section “Reference Times for Downtime during Import”. With a careful preparation you may reduce the impact of server downtime by incremental migration. For details see Section “Incremental Migration”.

UUIDs on Live Servers

Content UUIDs will be transferred to live servers upon publication automatically. Publishing newly imported content with UUIDs will thus transfer these UUIDs to the live servers as well. In a UUID migration between environments, you can apply the import file used for the Content Management Server to all live servers, too. See import guidelines for live servers in Section “Typical Usage” for details.

The first two tools content-uuid-export and generate-content-uuid-map do not require a downtime. Nevertheless, it is recommended to execute these tools during limited editorial activity. This will reduce a possible performance impact of content-uuid-export and the generated data may be more consistent. Otherwise, data may contain for example undesirable references to contents deleted concurrently.

Unlike content-uuid-export, you may execute generate-content-uuid-map without running Content Management Server. This may be convenient, if you want to run generate-content-uuid-map and content-uuid-import on Production Environment without interruption.

The last of these tools, content-uuid-import, requires a downtime of the Content Management Server in the Production Environment. If you have any clients running with automatic reconnect, these need to be shut down as well, as the change to UUIDs will not be propagated to them until a restart.

All three tools require direct access to the database. The tools use the configuration available in properties/corem/sql.properties. If these properties do not match the database to access, you may specify the database connection options via command line parameters like --dburl. Thus, if applicable, ensure that you have the following properties at hand:

See Section 3.2.4, “Properties for the Connection to the Database” in Deployment Manual for details on these properties.

Example Scenario: Live Servers Transfer

For the following example, a single CoreMedia Content Cloud instance with Content Management Server, Master Live Server, and one or more Replication Live Servers is expected.

Typical Usage

The following is a typical flow of actions when transferring UUIDs to live servers. Assuming that your Content Management Server is hosted at cms.host and server databases are hosted on cms.db.host, mls.db.host and rls.db.host, the following shows the order of commands. Repeat the last command for each Replication Live Server in your instance and adjust database connection accordingly. Remember that you need to shut down each live server before running the corresponding import command:

$ cm content-uuid-export \
    --user admin \
    --url https://cms.host/coremedia/ior \
    --query "BELOW PATH '/Sites' AND isPublished" \
    --dburl jdbc:mysql://cms.db.host:3306/cm_management \
    --dbuser cm_management \
    --dbpassword ... \
    --toimportformat \
    --output cms-export.csv

$ cm content-uuid-import \
    --dburl jdbc:mysql://mls.db.host:3306/cm_master \
    ...
    --input cms-export.csv

$ cm content-uuid-import \
    --dburl jdbc:mysql://rls.db.host:3306/cm_replication \
    ...
    --input cms-export.csv

Be extra careful when importing the file to any live server to make sure that you are only importing to a live server which has published content from the very Content Management Server that you created the export from. Importing to a live server which received content from another Content Management Server will introduce wrong UUIDs. When you discover that this happened accidentally, simply repeat the export on the correct Content Management Server and fix your damaged live server content with it by importing this correct export file.

Note, that you not only have to shut down your servers, but also restart any of its clients. This applies to clients with the ability to reconnect as well, as they will not get notified on changed UUIDs.

Preparation

To finish the process of UUID synchronization you will require a server downtime for live servers when using the import tool. For a rough estimation on possible downtimes see Section “Reference Times for Downtime during Import”. With a careful preparation you may reduce the impact of server downtime by incremental migration. For details see Section “Incremental Migration”.

The first tool, content-uuid-export, does not require a downtime. Nevertheless, it is recommended to execute the tool during limited editorial activity. This will reduce a possible performance impact and the generated data may be more consistent. Otherwise, data may contain for example undesirable references to contents unpublished concurrently.

The second tool, content-uuid-import, requires a downtime of the corresponding Master Live Server or Replication Live Server. If you have any clients running with automatic reconnect, these need to be shut down as well, as the change to UUIDs will not be propagated to them until a restart.

Both tools require direct access to the database. The tools use the configuration available in properties/corem/sql.properties. If these properties do not match the database to access, you may specify the database connection options via command line parameters like --dburl. Thus, if applicable, ensure that you have the following properties at hand:

See Section 3.2.4, “Properties for the Connection to the Database” in Deployment Manual for details on these properties.

Incremental Migration

An incremental migration approach increases the effort while decreasing the downtime of the servers and clients for one iteration. Having several options working with increments, choose the best of the options described in the following sections, or combine them. Whatever suits you best.

The strategies for incremental application apply to both scenarios alike (multi-environment migration and live servers transfer).

Incremental Export and Mapping

While content-uuid-export retrieves the content data by direct database access (due to performance reasons), you select the content to export by Unified API queries. For syntax and details see Section 5.7, “Query Service” in Unified API Developer Manual.

Split by Content Queries: It is recommended to split your migration tasks by paths. You may for example start with the root folder of your master site. Or you may start with everything outside of your sites folder. Using content queries provides a rich mechanism of partitioning your contents. For some examples on queries, consider calling content-uuid-export with --query-help.

Only when in a multi-environment migration scenario, you run generate-content-uuid-map with the results from content-uuid-export.

Incremental Import

The content-uuid-import provides the options --from and --to. These limit the processed elements provided by generate-content-uuid-map. --from specifies the first element to import. --to specifies the last element to import. If you run content-uuid-import with --from 10 and --to 20, elements 10 to 20 (inclusive) will be processed.

How can I ensure that all UUIDs have been imported correctly in a multi-environment migration scenario? The key to this is that generate-content-uuid-map will only output those mappings which require the UUID of a content to be changed. Thus, it ignores especially UUIDs which already match at Production Environment (for example from previous migration runs). Therefore, you could run generate-content-uuid-map again after your import. If this results in an empty mapping/file, all your UUIDs were mapped.

How to recover from aborted import? content-uuid-import can safely be aborted at any time. This, of course, also applies to unexpected abortions due to connection loss to database for example. To recover, you have several options, where the easiest one is just starting content-uuid-import again. This is because updating the UUID twice (to the same value) does no harm to the system. To skip already processed elements, you may either run generate-content-uuid-map again prior to starting content-uuid-import, or you may adjust the parameters --from and --to accordingly.

Reference Times for Downtime during Import

As stated before, while executing content-uuid-import the Content Server and related clients have to be stopped. This section will give you a rough overview on possible downtimes to expect for the last step.

Test Setup: A million contents were updated during the test. All databases were set up locally within Docker containers, each having memory of 5 GB and 8 CPUs. content-uuid-import was executed with --threads 8 and --batchsize 1000 (default).

Command Reference

In the following you will get detailed help on the specific commands in the order of their execution during the migration process.

content-uuid-export

content-uuid-export is the first tool to be executed. It will generate a CSV file (more specifically a semicolon-separated file, so that you may open it for example in several office applications directly). Without option --toimportformat , the CSV file will contain the following data:

Both path and type will be used to identify the content in Production Environment.

With option --toimportformat, the data contained in the CSV file will be in the format as the output of generate-content-uuid-map. See Section “generate-content-uuid-map” for details.

For exported contents that don't have a UUID yet, a new UUID will be created and persisted in the database. Creating a lot of missing UUIDs may severely slow down the export.

The options have the following meaning:

generate-content-uuid-map

generate-content-uuid-map is the tool to be executed in a multi-environment migration scenario. It will generate a CSV file for import (more specifically a semicolon-separated file, so that you may open it for example in several office applications directly). The CSV file will contain the following information, which are required to continue with the next task:

The ID will be used to identify the content in the database in Production Environment during the next task.

content-uuid-import

content-uuid-import is the last tool to be executed. It will update the contents of a Content Server to have the corresponding new UUIDs. In a multi-environment migration scenario, its input file will be the output of generate-content-uuid-map. In a live servers transfer scenario, its input file will be the output of content-uuid-export (called with option --toimportformat).

Prior to starting this tool, you need to shut down the target Content Management Server, Master Live Server, or Replication Live Server. To reduce downtimes, you may want to read Section “Incremental Migration”.