Synchronizing the configuration

The FGCP uses a combination of incremental and periodic synchronization to make certain that the configuration of all cluster units is synchronized to that of the primary unit.

The following settings are non synchronized between cluster units:

  • The FortiGate host name
  • GUI Dashboard widgets
  • HA override
  • HA device priority
  • The virtual cluster priority
  • The HA priority setting for a ping server (or dead gateway detection) configuration
  • The system interface settings of the HA reserved management interface
  • The HA default route for the reserved management interface, set using the ha-mgmt-interface-gateway pick of the config system ha command

Most subscriptions and licenses are not synchronized, as each FortiGate must exist licensed individually. FortiToken Mobile is an exception; they are registered to the primary unit and synchronized to the slaves.

The chief unit synchronizes all other configuration settings, including the other HA configuration settings.

All synchronization activity takes place over the HA heartbeat link using TCP/703 and UDP/703 packets.

Disabling automatic configuration synchronization

In some cases you may want to apply the following command to disable automatic synchronization of the primary unit configuration to all cluster units.

config system ha

set sync-config disable

end

When this option is disabled the cluster no longer synchronizes configuration changes. If a device failure occurs, the new primary unit may not take the same configuration as the failed primary unit. As a result, the new master unit of measurement may process sessions differently or may not function on the network in the same style.

In most cases you should not disable automatic configuration synchronization. Yet, if y'all take disabled this feature you can apply the execute ha synchronize command to manually synchronize a subordinate unit of measurement'due south configuration to that of the primary unit of measurement.

You lot must enter execute ha synchronize commands from the subordinate unit that you want to synchronize with the primary unit. Use the execute ha manage command to access a subordinate unit CLI.

For example, to access the first subordinate unit and strength a synchronization at any time, fifty-fifty if automated synchronization is disabled enter:

execute ha manage 0

execute ha synchronize start

You can utilize the following command to cease a synchronization that is in progress.

execute ha synchronize terminate

Incremental synchronization

When you log into the cluster GUI or CLI to make configuration changes, you are actually logging into the master unit. All of your configuration changes are outset made to the primary unit of measurement. Incremental synchronization and so immediately synchronizes these changes to all of the subordinate units.

When you log into a subordinate unit CLI (for case using execute ha manage) all of the configuration changes that y'all brand to the subordinate unit are also immediately synchronized to all cluster units, including the primary unit of measurement, using the same process.

Incremental synchronization likewise synchronizes other dynamic configuration information such as the DHCP server accost charter database, routing tabular array updates, IPsec SAs, MAC accost tables, and so on. Come across DHCP and PPPoE compatability for more information nigh DHCP server address charter synchronization and Synchronizing kernel routing tables for information about routing table updates.

Whenever a modify is fabricated to a cluster unit configuration, incremental synchronization sends the same configuration change to all other cluster units over the HA heartbeat link. An HA synchronization process running on the each cluster unit receives the configuration change and applies it to the cluster unit of measurement. The HA synchronization procedure makes the configuration change by entering a CLI command that appears to be entered by the administrator who fabricated the configuration change in the kickoff place.

Synchronization takes place silently, and no log messages are recorded nearly the synchronization activity. However, log letters can be recorded by the cluster units when the synchronization process enters CLI commands. Y'all can run across these log messages on the subordinate units if you enable event logging and set the minimum severity level to Information and so check the consequence log letters written by the cluster units when you lot make a configuration change.

You tin too see these log messages on the primary unit of measurement if yous brand configuration changes from a subordinate unit.

Periodic synchronization

Incremental synchronization makes sure that every bit an administrator makes configuration changes, the configurations of all cluster units remain the same. However, a number of factors could cause one or more than cluster units to get out of sync with the primary unit. For example, if yous add together a new unit to a performance cluster, the configuration of this new unit of measurement volition not match the configuration of the other cluster units. Its not applied to employ incremental synchronization to change the configuration of the new unit.

Periodic synchronization is a machinery that looks for synchronization problems and fixes them. Every minute the cluster compares the configuration file checksum of the main unit of measurement with the configuration file checksums of each of the subordinate units. If all subordinate unit of measurement checksums are the same equally the master unit checksum, all cluster units are considered synchronized.

If ane or more of the subordinate unit checksums is not the same as the main unit checksum, the subordinate unit configuration is considered out of sync with the primary unit. The checksum of the out of sync subordinate unit is checked over again every xv seconds. This re-checking occurs in instance the configurations are out of sync because an incremental configuration sequence has not completed. If the checksums do not match after v checks the subordinate unit that is out of sync retrieves the configuration from the primary unit. The subordinate unit then reloads its configuration and resumes operating as a subordinate unit of measurement with the same configuration as the primary unit.

The configuration of the subordinate unit is reset in this mode because when a subordinate unit configuration gets out of sync with the primary unit configuration in that location is no efficient way to determine what the configuration differences are and to correct them. Resetting the subordinate unit of measurement configuration becomes the most efficient style to resynchronize the subordinate unit of measurement.

Synchronization requires that all cluster units run the same FortiOS firmware build. If some cluster units are running different firmware builds, and then unstable cluster operation may occur and the cluster units may not be able to synchronize correctly.

note icon Re-installing the firmware build running on the primary unit forces the main unit of measurement to upgrade all cluster units to the same firmware build.

Panel messages when configuration synchronization succeeds

When a cluster commencement forms, or when a new unit is added to a cluster as a subordinate unit, the following letters announced on the CLI panel to indicate that the unit joined the cluster and had its configuring synchronized with the main unit.

slave'south configuration is not in sync with master'southward, sequence:0

slave'due south configuration is non in sync with master's, sequence:i

slave's configuration is non in sync with primary'southward, sequence:two

slave'due south configuration is not in sync with principal's, sequence:3

slave's configuration is non in sync with primary'due south, sequence:4

slave starts to sync with master

logout all admin users

slave succeeded to sync with master

Panel messages when configuration synchronization fails

If you connect to the console of a subordinate unit of measurement that is out of synchronization with the chief unit, messages similar to the post-obit are displayed.

slave is not in sync with primary, sequence:0. (type 0x3)

slave is not in sync with principal, sequence:1. (blazon 0x3)

slave is non in sync with master, sequence:2. (blazon 0x3)

slave is not in sync with master, sequence:iii. (type 0x3)

slave is not in sync with chief, sequence:four. (type 0x3)

global compared not matched

If synchronization problems occur the console message sequence may exist repeated over and over again. The letters all include a type value (in the instance type 0x3). The type value can help Fortinet Support diagnose the synchronization trouble.

HA out of sync object messages and the configuration objects that they reference
Out of Sync Bulletin Configuration Object
HA_SYNC_SETTING_CONFIGURATION = 0x03 /data/config
HA_SYNC_SETTING_AV = 0x10
HA_SYNC_SETTING_VIR_DB = 0x11 /etc/vir
HA_SYNC_SETTING_SHARED_LIB = 0x12 /data/lib/libav.and so
HA_SYNC_SETTING_SCAN_UNIT = 0x13 /bin/scanunitd
HA_SYNC_SETTING_IMAP_PRXY = 0x14 /bin/imapd
HA_SYNC_SETTING_SMTP_PRXY = 0x15 /bin/smtp
HA_SYNC_SETTING_POP3_PRXY = 0x16 /bin/pop3
HA_SYNC_SETTING_HTTP_PRXY = 0x17 /bin/thttp
HA_SYNC_SETTING_FTP_PRXY = 0x18 /bin/ftpd
HA_SYNC_SETTING_FCNI = 0x19 /etc/fcni.dat
HA_SYNC_SETTING_FDNI = 0x1a /etc/fdnservers.dat
HA_SYNC_SETTING_FSCI = 0x1b /etc/sci.dat
HA_SYNC_SETTING_FSAE = 0x1c /etc/fsae_adgrp.enshroud
HA_SYNC_SETTING_IDS = 0x20 /etc/ids.rules
HA_SYNC_SETTING_IDSUSER_RULES = 0x21 /etc/idsuser.rules
HA_SYNC_SETTING_IDSCUSTOM = 0x22
HA_SYNC_SETTING_IDS_MONITOR = 0x23 /bin/ipsmonitor
HA_SYNC_SETTING_IDS_SENSOR = 0x24 /bin/ipsengine
HA_SYNC_SETTING_NIDS_LIB = 0x25 /information/lib/libips.so
HA_SYNC_SETTING_WEBLISTS = 0x30
HA_SYNC_SETTING_CONTENTFILTER = 0x31 /data/cmdb/webfilter.bword
HA_SYNC_SETTING_URLFILTER = 0x32 /data/cmdb/webfilter.urlfilter
HA_SYNC_SETTING_FTGD_OVRD = 0x33 /data/cmdb/webfilter.fgtd-ovrd
HA_SYNC_SETTING_FTGD_LRATING = 0x34 /data/cmdb/webfilter.fgtd-ovrd
HA_SYNC_SETTING_EMAILLISTS = 0x40
HA_SYNC_SETTING_EMAILCONTENT = 0x41 /information/cmdb/spamfilter.bword
HA_SYNC_SETTING_EMAILBWLIST = 0x42 /data/cmdb/spamfilter.emailbwl
HA_SYNC_SETTING_IPBWL = 0x43 /data/cmdb/spamfilter.ipbwl
HA_SYNC_SETTING_MHEADER = 0x44 /data/cmdb/spamfilter.mheader
HA_SYNC_SETTING_RBL = 0x45 /data/cmdb/spamfilter.rbl
HA_SYNC_SETTING_CERT_CONF = 0x50 /etc/cert/cert.conf
HA_SYNC_SETTING_CERT_CA = 0x51 /etc/cert/ca
HA_SYNC_SETTING_CERT_LOCAL = 0x52 /etc/cert/local
HA_SYNC_SETTING_CERT_CRL = 0x53 /etc/cert/crl
HA_SYNC_SETTING_DB_VER = 0x55
HA_GET_DETAIL_CSUM = 0x71
HA_SYNC_CC_SIG = 0x75 /etc/cc_sig.dat
HA_SYNC_CC_OP = 0x76 /etc/cc_op
HA_SYNC_CC_MAIN = 0x77 /etc/cc_main
HA_SYNC_FTGD_CAT_LIST = 0x7a /migadmin/webfilter/ublock/ftgd/ data/

Comparing checksums of cluster units

Y'all tin use the diagnose sys ha checksum testify command to compare the configuration checksums of all cluster units. The output of this command shows checksums labeled global and all too as checksums for each of the VDOMs including the root VDOM. The get system ha-nonsync-csum control can be used to display similar information; however, this command is intended to be used by FortiManager.

The chief unit and subordinate unit of measurement checksums should exist the same. If they are non you can apply the execute ha synchronize commencement command to strength a synchronization.

The following control output is for the primary unit of a cluster that does non accept multiple VDOMs enabled:

diagnose sys ha checksum show

is_manage_master()=i, is_root_master()=1

debugzone

global: a0 7f a7 ff air conditioning 00 d5 b6 82 37 cc thirteen 3e 0b 9b 77

root: 43 72 47 68 7b da 81 17 c8 f5 10 dd fd 6b e9 57

all: c5 xc ed 22 24 3e 96 06 44 35 b6 63 7c 84 88 d5

checksum

global: a0 7f a7 ff ac 00 d5 b6 82 37 cc thirteen 3e 0b 9b 77

root: 43 72 47 68 7b da 81 17 c8 f5 10 dd fd 6b e9 57

all: c5 90 ed 22 24 3e 96 06 44 35 b6 63 7c 84 88 d5

The following command output is for a subordinate unit of the same cluster:

diagnose sys ha checksum prove

is_manage_master()=0, is_root_master()=0

debugzone

global: a0 7f a7 ff ac 00 d5 b6 82 37 cc thirteen 3e 0b 9b 77

root: 43 72 47 68 7b da 81 17 c8 f5 ten dd fd 6b e9 57

all: c5 90 ed 22 24 3e 96 06 44 35 b6 63 7c 84 88 d5

checksum

global: a0 7f a7 ff air conditioning 00 d5 b6 82 37 cc 13 3e 0b 9b 77

root: 43 72 47 68 7b da 81 17 c8 f5 10 dd fd 6b e9 57

all: c5 90 ed 22 24 3e 96 06 44 35 b6 63 7c 84 88 d5

The following instance shows using this command for the master unit of a cluster with multiple VDOMs. 2 VDOMs have been added named test and Eng_vdm.

From the primary unit of measurement:

config global

diagnose sys ha checksum prove

is_manage_master()=1, is_root_master()=1

debugzone

global: 65 75 88 97 2d 58 1b bf 38 d3 3d 52 5b 0e 30 a9

examination: a5 xvi 34 8c 7a 46 d6 a4 1e 1f c8 64 ec 1b 53 iron

root: 3c 12 45 98 69 f2 d8 08 24 cf 02 ea 71 57 a7 01

Eng_vdm: 64 51 7c 58 97 79 b1 b3 b3 ed 5c ec cd 07 74 09

all: 30 68 77 82 a1 5d 13 99 d1 42 a3 2f 9f b9 15 53

checksum

global: 65 75 88 97 2d 58 1b bf 38 d3 3d 52 5b 0e xxx a9

examination: a5 16 34 8c 7a 46 d6 a4 1e 1f c8 64 ec 1b 53 fe

root: 3c 12 45 98 69 f2 d8 08 24 cf 02 ea 71 57 a7 01

Eng_vdm: 64 51 7c 58 97 79 b1 b3 b3 ed 5c ec cd 07 74 09

all: 30 68 77 82 a1 5d 13 99 d1 42 a3 2f 9f b9 15 53

From the subordinate unit:

config global

diagnose sys ha checksum bear witness

is_manage_master()=0, is_root_master()=0

debugzone

global: 65 75 88 97 2d 58 1b bf 38 d3 3d 52 5b 0e 30 a9

exam: a5 16 34 8c 7a 46 d6 a4 1e 1f c8 64 ec 1b 53 atomic number 26

root: 3c 12 45 98 69 f2 d8 08 24 cf 02 ea 71 57 a7 01

Eng_vdm: 64 51 7c 58 97 79 b1 b3 b3 ed 5c ec cd 07 74 09

all: 30 68 77 82 a1 5d thirteen 99 d1 42 a3 2f 9f b9 xv 53

checksum

global: 65 75 88 97 2nd 58 1b bf 38 d3 3d 52 5b 0e xxx a9

exam: a5 16 34 8c 7a 46 d6 a4 1e 1f c8 64 ec 1b 53 atomic number 26

root: 3c 12 45 98 69 f2 d8 08 24 cf 02 ea 71 57 a7 01

Eng_vdm: 64 51 7c 58 97 79 b1 b3 b3 ed 5c ec cd 07 74 09

all: 30 68 77 82 a1 5d 13 99 d1 42 a3 2f 9f b9 15 53

How to diagnose HA out of sync messages

This section describes how to employ the diagnose sys ha checksum show and diagnose debug commands to diagnose the cause of HA out of sync letters.

If HA synchronization is not successful, employ the post-obit procedures on each cluster unit of measurement to find the cause.

To decide why HA synchronization does not occur
  1. Connect to each cluster unit CLI by connected to the console port.

  2. Enter the following commands to enable debugging and brandish HA out of sync messages.

    diagnose debug enable

    diagnose debug console timestamp enable

    diagnose debug awarding hatalk -1

    diagnose debug application hasync -1

    Collect the console output and compare the out of sync messages with the information in the table HA out of sync object letters and the configuration objects that they reference.

  3. Enter the following commands to turn off debugging.

    diagnose debug disable

    diagnose debug reset

To make up one's mind what part of the configuration is causing the problem

If the previous process displays messages that include sync object 0x30 (for example, HA_SYNC_SETTING_CONFIGURATION = 0x03) there is a synchronization problem with the configuration. Utilize the following steps to decide the function of the configuration that is causing the trouble.

If your cluster consists of two cluster units, use this process to capture the configuration checksums for each unit of measurement. If your cluster consists of more that two cluster units, repeat this procedure for all cluster units that returned messages that include 0x30 sync object letters.

  1. Connect to each cluster unit of measurement CLI past connected to the panel port.

  2. Enter the following command to turn on terminal capture

    diagnose debug enable

  3. Enter the following command to stop HA synchronization.

    execute ha sync terminate

  4. Enter the following command to display configuration checksums.

    diagnose sys ha checksum show global

  5. Re-create the output to a text file.
  6. Repeat for all affected units.

  7. Compare the text file from the primary unit with the text file from each cluster unit of measurement to find the checksums that do not match.

    Yous tin can use a diff office to compare text files.

  8. Repeat for the root VDOM:

    diagnose sys ha checksum show root

  9. Repeat for all VDOMS (if multiple VDOM configuration is enabled):

    diagnose sys ha checksum show <vdom-name>

  10. You can also apply the grep option to merely display checksums for parts of the configuration.

    For example to brandish organisation related configuration checksums in the root VDOM or log-related checksums in the global configuration:

    diagnose sys ha checksum root | grep system

    diagnose sys ha chechsum global | grep log

    More often than not it is the commencement non-matching checksum that is the crusade of the synchronization problem.

  11. Attempt to remove/change the role of the configuration that is causing the problem. You can practice this by making configuration changes from the principal unit or subordinate unit CLI.

  12. Enter the post-obit commands to start HA configuration and stop debugging:

    execute ha sync starting time

    diagnose debug disable

    diagnose debug reset

Recalculating the checksums to resolve out of sync messages

Sometimes an error tin occur when checksums are being calculated by the cluster. As a event of this adding fault the CLI console could display out of sync error messages fifty-fifty though the cluster is otherwise operating normally. You can likewise sometimes see checksum adding errors in diagnose sys ha checksum control output when the checksums listed in the debugzone output don't lucifer the checksums in the checksum role of the output.

One solution to this problem could be to re-calculate the checksums. The re-calculated checksums should friction match and the out of sync error letters should stop appearing.

You can employ the following command to re-calculate HA checksums:

diagnose sys ha checksum recalculate [<vdom-proper name> | global]

Only entering the command without options recalculates all checksums. Yous can specify a VDOM proper noun to but recalculate the checksums for that VDOM. You can too enter global to recalculate the global checksum.

Determining what is causing a configuration synchronization problem

There are twenty-v FortiOS modules that accept their configurations synchronized. It can be difficult to observe the cause of a synchronization problem with and then much data to analyze. Y'all tin can use the post-obit diagnose commands to more easily find modules that may be causing synchronization bug.

diagnose sys ha hasync-stats {all | well-nigh-recent [<seconds>] | by object [<number>]}

all displays the synchronization action for all modules that happened since the hasync process started running (usually this would be since the cluster started-up).

most-contempo [<seconds>] displays the virtually recently occurring synchronization events. You tin include a time in seconds to display recent events that occurred during the fourth dimension interval. If you don't include the number of seconds, the control displays the most recent events in the last 5 seconds. This option tin can be used to decide the module or modules that are currently synchronizing or attempting to synchronize. If no modules are currently synchronizing, the command but displays the virtually recent synchronization events.

past-object [<number>] displays the synchronization activity of a specific module, where <number> is the module number in the range one to 25. To display a list of all 25 modules and their numbers enter:

diagnose sys ha hasync-stats by-object ?

To display the well-nigh recent activity, enter:

diagnose sys ha hasync-stats most-recent x electric current-fourth dimension/jiffies=2018-03-28 13:01:42/1148242:  hasync-obj=2(arp):     epoll_handler=1(ev_arp_handler): outset=1522256500.354400(2018-03-28 13:01:40), end=1522256500.354406(2018-03-28 13:01:40), total=0.000006/1699 hasync-obj=five(config):     timer=0(check_sync_status), add=1141764(2018-03-28 13:01:26), expire=1142764(2018-03-28 13:01:36), finish=1142764(25018-03-28 xiii:01:36), del=0(), total_call=1143 hasync-obj=8(time):     obj_handler=0(packet): start=1522256497.851550(2018-03-28 13:01:37), end=1522256497.851570(2018-03-28 13:01:37), total=0.000020/381     timer=0(sync_timer), add=1140106(2018-03-28 13:01:10), expire=1143106(2018-03-28 13:01:40), cease=1143106(2018-03-28 13:01:twoscore), del=0(), total_call=381 hasync-obj=21(hastats):     obj_handler=0(packet): first=1522256499.760934(2018-03-28 xiii:01:39), end=1522256499.760936(2018-03-28 thirteen:01:39), full=0.000002/2285     timer=0(hastats_timer), add=1142556(2018-03-28 13:01:34), expire=1143056(2018-03-28 xiii:01:39), end=1143056(2018-03-28 thirteen:01:39), del=0(), total_call=2286

The terminal few lines of this output shows activity with the hastats module, which is module 21. You can employ the post-obit control to see more information near synchronization activity with this module:

diagnose sys ha hasync-stats by-object 21