Network Fun!!! -- A Security/Network Engineer's Blog: Magic Mac Address: What To Do When You Have Multiple Check Point Clusters On The Same Subnet OR In Parallel With Each Other

(NOTE*** Please read through this whole posting. This isn't one of those posts where you can just go to the config and get the answer right away. Thanks.) Also, this was a two man team effort. Thanks Chris.

See the below topology. Because the clusters by default have the same MAC address for the virtual IP address, and because the two clusters are in parallel with each other, you have to change one of the cluster MAC addresses so that they are not the same. If you do not, then the upstream router will wig out and cause you all kinds of instability issues. Think about how ARP works, then think about if you had two MAC addresses that were the same on the network. You can see that that's a big no no.

[Expert@CPfirewall1]# cphaconf set_ccp broadcast <---- You can change this mode, but multicast is the default and recommended.

[Expert@CPfirewall1]# cphaconf set_ccp multicast <---- Multicast is the default, you prefer this.

[Expert@CPfirewall1]# CPfirewall ctl set int CPfirewallha_mac_magic 251 <---- (251 in Decimal is FB in Hex)(CCP traffic)

[Expert@CPfirewall1]# CPfirewall ctl set int CPfirewallha_mac_forward_magic 250 <---- (250 in Decimal FA in Hex)(Forwarding Layer traffic)

Expert@CPfirewall1]# CPfirewall ctl get int CPfirewallha_mac_magic <---- This command tells you what the value is set to for "CPfirewall ctl set int CPfirewallha_mac_magic"

CPfirewallha_mac_magic = 251 <---- (251 in Decimal FA in Hex)(CCP traffic)

[Expert@CPfirewall1]# CPfirewall ctl get int CPfirewallha_mac_forward_magic <---- This command tells you what the value is set to for "CPfirewall ctl set int CPfirewallha_mac_forward_magic"

CPfirewallha_mac_forward_magic = 250 <---- (250 in Decimal is FB in Hex)(Forwarding Layer traffic)

[Expert@CPfirewall1]#

The defaults for the Cluster MAC address is as follows:

For (CCP traffic): "fwha_mac_magic=0xfe" OR "CPfirewall ctl set int CPfirewallha_mac_magic 254"

For (Forwarding Layer traffic): "fwha_mac_forward_magic=0xfd" OR "CPfirewall ctl set int CPfirewallha_mac_forward_magic 253"

Now, one question I have had is this: Are the above commands the same? Meaning, do "fwha_mac_magic=0xfe" OR "CPfirewall ctl set int CPfirewallha_mac_magic 254" mean the same thing? It does appear that they are.

So I think I can conclude also that these two are the same as well: "fwha_mac_forward_magic=0xfd" OR "CPfirewall ctl set int CPfirewallha_mac_forward_magic 253"

So from what I can find, these two for each set should be the same. Especially after what we read next.

-------------

(addition...)

So I contacted Check Point TAC to verify that the two sets of commands above were actually the same. This is what they responded back to me with via email:

(email...)

The valid kernel parameters to change the magic Mac are:

fwha_mac_magic 
fwha_mac_forward_magic

And the procedure should be:
On each of the Cluster Modules
1. cd $FWDIR/boot/modules
2. create the fwkern.conf file by: # vi fwkern.conf
3. Add the required parameters and values as given below:
fwha_mac_magic = 250
fwha_mac_forward_magic = 251
4. Save the fwkern.conf
5. Verify the fwker.conf is correctly configured by: # more fwkern.conf
6. Reboot the Module
7. Verify the new mac magic setups correctly configured by:
# fw ctl get int fwha_mac_magic
# fw ctl get int fwha_mac_forward_magic
8. Verify the Cluster Module status by:
# cphaprob stat
** the 250/251 should be the SAME on both cluster member , but should be DIFFERENT for each different cluster **
We do not recognize the second parameter, if you think it is relevant please let me know where you took it from.

(END email...)

Ok, to me, this TAC answer seems different than what the Check Point sk36913 says.  Below is a cut and paste from that sk...

---------------------- (BEGIN of sk36913) ----------------------

Cause

The Cluster Control Protocol (CCP) packets that are sent between the members of the same cluster reach the neighbor cluster (connected to the same network) and "confuse" it.

Solution

The "confusion" rises from the fact that CCP packets in Check Point clusters have the same Source MAC address - 00:00:00:00:mac_magic:member_id (where - mac_magic (5th field) designates the type of CCP packet, and member_id (6th field) designates the Member ID).

Refer to any version ClusterXL Administration Guide -> Chapter ClusterXL Advanced Configuration -> Working with VLANS and Clusters -> Connecting Several Clusters on the Same VLAN

The 5th field of Source MAC Address - mac_magic - has to be unique for each cluster.
This is controlled with the following kernel parameters :
fwha_mac_magic
fwha_mac_forward_magic

To get the values of these parameters, run:

# fw ctl get int fwha_mac_magic
# fw ctl get int fwha_mac_forward_magic
Note: in this case, the numbers will be shown in Decimal format

Default values on non-VSX : fwha_mac_magic = 254 , fwha_mac_forward_magic = 253
Default values on VSX : fwha_mac_magic = 246 , fwha_mac_forward_magic = 245

To set the values of these parameters on-the-fly:

# fw ctl set int fwha_mac_magic YY
# fw ctl set int fwha_mac_forward_magic ZZ

Note: YY and ZZ have to be given in Decimal format
Example :
# fw ctl set int fwha_mac_magic 57
# fw ctl set int fwha_mac_forward_magic 56

To set the values of these parameters permanently, refer to sk26202 (I reference this below)

Note: the YY and ZZ should be set in Decimal format; if the values are not accepted after reboot, then set them in Hexadecimal format (like 0xYY and 0xZZ)

---------------------- (END of sk36913) ----------------------

Ok, so make matters a little more confusing, here is another sk about this same issue. Its sk25977. See below the cut and paste for that sk. By the way, didnt that TAC engineer say in their email to me that they didnt recognize the commands in sk36913 (above)???

---------------------- (BEGIN of sk25977) --------------------

Cause

The Cluster Control Protocol (CCP) packets that are sent between the members of the same cluster reach the neighbor cluster (connected to the same network) and "confuse" it

Solution

It is not recommended to connect interfaces of multiple clusters to the same network segment (same VLAN, same switch). A separate VLAN and/or switch is recommended for each cluster. A crossover link may be used for the sync (secured) interfaces.

If there is a need to connect the interfaces (secured or non-secured) of multiple clusters to the same network segment, you need to make changes to:

The "Source MAC address" of the Cluster Control Protocol, to enable communication between cluster members.
(For all ClusterXL modes, both High Availability and Load Sharing).

The "Destination MAC address", to enable communication between the cluster and machines outside the cluster.
(For ClusterXL Load Sharing Multicast Mode clusters only).

Steps to change Source MAC Addresses:
(For all ClusterXL modes, both High Availability and Load Sharing)

How the Source Cluster MAC Address is assigned:
Cluster members communicate with each other using the Cluster Control Protocol (CCP). CCP packets are distinguished from ordinary network traffic by giving CCP packets a unique Source MAC address.

The first four bytes of the source MAC address are all zero: 00:00:00:00
The 5th byte of the source MAC address is a 'magic number'. Its value indicates its purpose. The default value is:
CCP traffic - 0xFE on ClusterXL , 0xF6 on VSX cluster
Forwarding Layer traffic - 0xFD , 0xF5 on VSX cluster
The 6th byte is the ID of the sending cluster member.

PROBLEM OVERVIEW (Duplicate Source Cluster MAC Addresses):
When more than one cluster is connected to the same VLAN, if CCP and forwarding layer traffic uses multicast, this traffic reaches only the intended cluster.
If CCP and forwarding layer traffic (and in certain other cases) use broadcast, cluster traffic intended for one cluster is seen by all connected clusters, and is processed by the wrong cluster, which causes communication problems.

SOLUTION:
To ensure that the source MAC address in packets coming from different clusters that are connected to the same VLAN can be distinguished, change the source MAC address of the cluster interface that is connected to the VLAN in all but one of the clusters. Configure the following security gateway parameters to set more than one cluster on the same VLAN (these parameters apply to both ClusterXL and OPSEC certified clustering products. Note that CCP traffic is transmitted only through the Sync interface of OPSEC certified clustering products:

Module parameters with default values of 5th byte:
fwha_mac_magic=0xfe (CCP traffic)
fwha_mac_forward_magic=0xfd (Forwarding Layer traffic)

These parameters apply to both ClusterXL and OPSEC certified clustering products. Changing the values of these module configuration parameters alters the fifth part of the source MAC address of Cluster Control Protocol and forwarded packets. Use any value as long as the two module configuration parameters are different. To avoid confusion, do not use the value 0x00 or 0xFF.

For example you may want to configure these parameters as the following:

fwha_mac_magic=0xfb 
fwha_mac_forward_magic=0xfa

Note:
When Performance Pack is used to enhance the performance of ClusterXL Load Sharing Multicast Mode, it is recommended that the chosen numbers be consecutive, with the lower one being even (e.g., 0x10 and 0x11, or 0xBE and 0xBF, in the example above 'a' and 'b').

PROCEDURE (configuration surviving reboot per sk26202):
To be performed on each cluster member that is connected to the same network segment.
Very important: when editing multiple clusters, please make sure that values are different (values should be same for 2 members of one cluster), for example 'fe' and 'fd' for same cluster members, but for different cluster it would be i.e., 'ba' and 'bb' values.

Steps to change Destination Multicast MAC Addresses:
(For ClusterXL Load Sharing Multicast Mode clusters only)

PROCEDURE (For user-defined multicast MAC address):

In the ClusterXL page of the cluster object, select Load Sharing Multicast Mode. In the 'Topology' tab, edit the cluster interface that is connected to same network segment as other cluster(s).
In the 'Interface Properties' window - 'General' tab - click 'Advanced'.
Change the default MAC address, and carefully type the new user-defined MAC address. It must be of the form 01:00:5e:xy:yy:yy, where X is between 0 and 7, and Y is between 0 and F (Hex).
Click 'OK'.
Install the Security Policy onto the cluster.

---------------------- (END of sk25977) --------------------

OK, FINAL CONCLUSION: I sent an email to a couple of technical engineers at Check Point that I know. I point blank asked them which one is it that is accurate. One engineer said this, among other things: "I think what the TAC sent you in the included email is probably as good as anything to follow since it includes the method for making it a permanent change."

I trust the engineer enough to go with what he says.

Now, just one more confusing note in case you see it.  In sk26202, it says the following to make the change permanent.  It was referenced in sk36913 (above).  I tried this below, it didnt work for me.

------------------(BEGIN sk26202)--------------

Solution

Global kernel parameters exist to control (customize) the behavior of Security Gateway (kernel parameters are located in $FWDIR/boot/modules/fw*mod* kernel modules).
This control (customization) can be done on-the-fly using the fw ctl set int command (change takes effect immediately). However, the value of the kernel parameter returns to its default value after a reboot. At times, it may be required to control (customize) the behavior of Security Gateway permanently. In addition, it is necessary for some kernel parameters to be changed upon boot.

Setting kernel global parameters on-the-fly is the same on all OS's using the fw ctl set int command.
Warning: applies to Security Gateways ONLY.
Note: Verify the existence of the kernel parameter first using fw ctl get int command.
Example: fw ctl set int fwseqvalid_exact_syn_on_rst 2
Setting kernel global parameters permanently is unique for some OS's.
Warning: applies to Security Gateways ONLY.
Note: Verify the existance of the kernel parameter parameter first using fw ctl get int command.
Example: fw ctl get int fwseqvalid_exact_syn_on_rst

Note: For SecureXL global kernel parameters (located in $PPKDIR/boot/modules/sim*mod* kernel modules) refer to sk43387.

------------------(END sk26202)--------------

***SOLUTION UPDATE:  I did what the TAC engineer said to do in the email above, with exception of one thing.  I put in the HEX number instead of the DECIMAL number.  What I did worked, and it now shows up correctly in the ARP table of my Cisco switch. Below, in yellow, is what I changed on my second cluster.  In orange, is the default original cluster.

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
   2    0000.0000.fe00    DYNAMIC     Gi1/0/13
   2    0000.0000.fe01    DYNAMIC     Gi1/0/14
   1    0000.0000.fb00    DYNAMIC     Gi1/0/8
   1    0000.0000.fb01    DYNAMIC     Gi1/0/7

Now, when I hook this up and verify everything is all as it should be, I see the following on the Cisco switch:

Switch#sh mac-address-table dyn
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports        ***NOTES***
----    -----------       --------    -----
   2    0000.0000.fe00    DYNAMIC     Gi1/0/13    (Sync - Cluster#1 Device#1 Virtual MAC)
   2    0000.0000.fe01    DYNAMIC     Gi1/0/14    (Sync - Cluster#1 Device#2 Virtual MAC)
   2    0091.5yh8.9e4b    DYNAMIC     Gi1/0/14    (Sync - Cluster#1 Device#2 Real MAC)
   2    0091.5yh8.a19b    DYNAMIC     Gi1/0/13    (Sync - Cluster#1 Device#1 Real MAC)
   1    0000.0000.fb00    DYNAMIC     Gi1/0/8      (Public - Cluster#2 Device#1 Virtual MAC)
   1    0000.0000.fb01    DYNAMIC     Gi1/0/7      (Public - Cluster#2 Device#2 Virtual MAC)
   1    0000.0000.fe00    DYNAMIC     Gi1/0/23    (Public - Cluster#1 Device#1 Virtual MAC)
   1    0000.0000.fe01    DYNAMIC     Gi1/0/24    (Public - Cluster#1 Device#2 Virtual MAC)
   1    002a.6r4t.104d    DYNAMIC     Gi1/0/7      (Public - Cluster#2 Device#2 Real MAC)
   1    002a.6r4t.10c5    DYNAMIC     Gi1/0/8      (Public - Cluster#2 Device#1 Real MAC)
   1    0091.5yh8.a194    DYNAMIC     Gi1/0/23   (Public - Cluster#1 Device#1 Real MAC)
   3    0000.0000.fa00    DYNAMIC     Gi1/0/15   (Sync - Cluster#2 Device#1 Virtual MAC for CCP Traffic)
   3    0000.0000.fb00    DYNAMIC     Gi1/0/16   (Sync - Cluster#2 Device#2 Virtual MAC)
   3    0000.0000.fb01    DYNAMIC     Gi1/0/15   (Sync - Cluster#2 Device#1 Virtual MAC)
   3    001c.6r4t.08ae    DYNAMIC     Gi1/0/16   (Sync - Cluster#2 Device#2 Real MAC)
   3    001c.6r4t.0932    DYNAMIC     Gi1/0/15   (Sync - Cluster#2 Device#1 Real MAC)

Network Fun!!! -- A Security/Network Engineer's Blog

Pages

Sunday, July 15, 2012

Magic Mac Address: What To Do When You Have Multiple Check Point Clusters On The Same Subnet OR In Parallel With Each Other