The replicator component is a Replication Live Server. It can be started and stopped while the server itself continues running. It consists of a controller process and a number of stages that process events for the Master Live Server. Events are always processed in the order in which they were created.

The replicator depends on the availability of two servers, two databases, and a network connection. The replication fails to make progress if any of these components fails. This section gives some hints for analyzing the replicator state if you suspect that events from the Master Live Server are not properly replayed on the Replication Live Server.

Checking the Server States

Using

cm runlevel -u <user> -p <password>
cm systeminfo -u admin

you can verify whether both servers are up and in runlevel online. The system information also tells you whether the initial replication of the Replication Live Server has been completed successfully.

An alternative way to check the replication service status is by querying the actuator HTTP endpoint as follows:

 curl -X GET http://localhost:42081/actuator/replicator

Checking the Replicator Configuration

Look at the IOR URL of the Master Live Server in the property replicator.publicationIorUrl and at the property replicator.enable, which starts and stops the replicator. In case of an error that is not automatically healed for an extended period, it can make sense to set replicator.autoRestart=false to ensure that the error condition can be analyzed without continuous restart attempts of the replicator.

Checking Replicator Startup Messages

If in doubt whether and when the replicator was started, check the log for messages of the form

[CURRENT_DATE] Info: cap.server.replicator:
Action(name="StartAction", completed=false): running
[CURRENT_DATE] Info: cap.server.replicator:
Replicator: pipeline created
...
[CURRENT_DATE] Info: cap.server.replicator:
Replicator: connected
[CURRENT_DATE] Info: cap.server.replicator:
Action(name="StartAction", completed=true): completed

where CURRENT_DATE is the date when the replicator was started, typically shortly after the server start. If these messages are repeated over and over again, the connection to the Master Live Server might be broken, especially if only the pipeline created message is printed, but the replicator never claims to be connected.

Checking Replicator Status Messages

After a successful start, the replicator writes frequent status messages to its log file on the log facility cap.server.replicator at log level info. A healthy idle replicator looks like this:

[CURRENT_DATE] Info: cap.server.replicator: EventStatistics: 
Replicator(enabled=true, initialized=false, state="running", 
pipelineUp=true, connectionUp=true, logEvents=false,
autoRestart=true, checkStream=true, checkTimeout=300)

[CURRENT_DATE] Info: cap.server.replicator: EventStatistics: 
IncomingCounter(lastSequenceNumber=SEQ_NO, 
lastStampedNumber=SEQ_NO, count=EVENT_COUNT, 
lastEventArrived=LAST_EVENT_DATE, 
startedAt=REPLICATOR_START_DATE)

[CURRENT_DATE] Info: cap.server.replicator: EventStatistics: 
CompletedCounter(lastSequenceNumber=SEQ_NO, 
lastStampedNumber=SEQ_NO, count=EVENT_COUNT, 
lastEventArrived=LAST_EVENT_DATE, 
startedAt=REPLICATOR_START_DATE)

where REPLICATOR_START_DATE is a date very close to the start of the Replication Live Server or the last start of the replicator, if the replicator has been restarted since the server start. LAST_EVENT_DATE should be a date very close to the last successful publication or the last replicator start, which ever comes later. SEQ_NO should be the sequence number of the last event received or 0, if no content has been published since the replicator start.

If messages of this type do not appear once per minute, check the log configuration and use the above mentioned command to check the server state. Make sure the replicator is enabled in the file replicatior.properties.

The three entries in the log have the following meaning:

The first line tells you to which extent the replicator is up. It also provides some basic configuration information:

The second line reports on the incoming events from the Master Live Server. When a publication is performed, the reported values should quickly rise to the last sequence number reported at the Master Live Server by cm events.

The third line reports the complete processing of events by the replicator pipeline. The reported properties are identical to the properties reported by the incoming event counter. Normally, the values reported here should lag only slightly behind the properties for the incoming events or should match them exactly. However, there are legitimate reasons for differences:

Generally, you should not worry about the health of the replicator as long as the property count of the processed events is continually rising, because that indicates that events are still being processed.

Interpreting Special Messages

The replicator outputs quite a lot of log messages in special occasions. The most frequent messages will be discussed.