High Availability Scenarios With IBM Tivoli Workload Scheduler And IBM Tivoli Framework
|
| < Day Day Up > |
|
In this section we offer our observations, together with questions and answers related with our implementation.
Observation 1
HACMP startup does not occur until both cluster nodes are running HACMP. After rebooting both nodes, we started the HACMP services on Node01 first and checked whether IBM Tivoli Workload Schedule had started. But after 10 or 15 minutes, IBM Tivoli Workload Scheduler still had not started.
After waiting for some time, we started the HACMP services on Node02. Shortly after Node02 started its HACMP services, we saw the IBM Tivoli Workload Scheduler application come up successfully on Node01. We have placed UNIX "wall" commands in the IBM Tivoli Workload Scheduler startup (and shutdown) scripts, so we will see exactly when these IBM Tivoli Workload Scheduler-related scripts are invoked.
| Q: | Our environment is a two-node cluster dedicated to running the IBM Tivoli Workload Scheduler process tree (the second node in the cluster sits idle). Therefore, wouldn't it make sense for us to start the HACMP IBM Tivoli Workload Scheduler Resource Group as soon as possible, regardless of which node comes up first? | |
| Q: | If this is acceptable (and advisable), exactly how is the HACMP configuration modified to accomplish this goal? | |
Answers
| A: | Yes, and that is normal. Your cascading config, as far as having node priority, is listed to have it start on Node01. |
| A: | Why you are dependent on the second node to start should be related to how your NFS is set up. You may leave your fs as a local mount and export it, but do not nfs mount it back to itself. |
Observation 2
During the startup of the HACMP Cluster, the connection to Node01 is lost. What occurs during this procedure is that the IP address on the Ethernet adapter is migrated to the EMEAMDM Service address (9.149.248.77). During this migration, your connection is broken and you must now physically reconnect to the machine through the EMEAMDM address.
| Q: | Does the addition of a third IP/Address (IP Aliasing) resolve this issue? | |
| Q: | Would this third IP address require an additional Ethernet adapter? | |
| Q: | Would this third IP address need to be in a different subnet from the other two addresses? | |
Answers
| A: | Yes. Your setup what is called a node alias, and probably even changed your topology config, where the boot and standby are both boot adapters. This would implement IP address takeover via aliasing (which would also be fast). However, node alias itself may not resolve this if it comes up on the boot adapter, which we believe is normal. So we think you would want to implement both node alias and IPAT via aliasing. |
| A: | No, it does not. |
| A: | Yes, it would. Here is what to do: Change your standby adapter to be a type "boot" adapter, and change your existing boot adapter(s) to be a different subnet than your service adapter subnet. This will give you a total of three subnets being used. |
Then you can create a node alias, which can be the same subnet as the service, and it is actually quite normal to do so.
Figure A-5 shows a generic example of topology config with IPAT via aliasing and the node alias, which is listed as persistent. This configuration requires a minimum of three subnets. The persistent address and service addresses can be on the same subnet (which is normal) or on separate subnets. This is also true when using multiple service addresses.
(This example shows mutual takeover, which means node B fails to A also, so the service 1b does not apply for you, but should give you the idea.)
Observation 3
During the failover process from Node01 to Node02, the service address on Node02 (9.149.248.74) remains unchanged, while the standby adapter (EN2 - 9.149.248.114) is migrated to the EMEAMDM service address (9.149.248.77). (In contrast, when HACMP services are started, we do get disconnected from the primary adapter in Node01, which is what we expected.)
In this configuration, when we telnet into the EN1 adapters (9.149.248.72 and 9.149.248.74) on both machines, we do not get disconnected from the machine during the failover process.
| Q: | Is this behavior expected (or desired)? | |
| Q: | Is this situation something we would like to see our Node01 do? (For example, have the secondary adapter (EN3) switch over to the EMEAMDM Service address, while EN2 (9.149.248.72) remains untouched and essentially acts as the backup Ethernet adapter.) | |
Answers
| A: | This is normal when doing traditional IPAT and one-sided takeover, because fallover of a service address will always move to the standby adapter, either locally for NIC failure, or remotely on system failure. If you implemented aliasing, you would not see any significant difference. |
| A: | You could see the desired results if you implement aliasing. |
Observation 4
Upon starting the HACMP Services on the nodes, we see content like that shown in Example A-4 in our smit logs.
Example A-4: smit logs
Oct 17 2003 20:56:39Starting execution of /usr/es/sbin/cluster/etc/rc.cluster with parameters: -boot -N -b 0513-029 The portmap Subsystem is already active. Multiple instances are not supported. 0513-029 The inetd Subsystem is already active. Multiple instances are not supported. Oct 17 2003 20:56:51Checking for srcmstr active...Oct 17 2003 20:56:51complete. 23026 - 0:00 syslogd Oct 17 2003 20:56:52 /usr/es/sbin/cluster/utilities/clstart : called with flags -sm -b 0513-059 The topsvcs Subsystem has been started. Subsystem PID is 20992. 0513-059 The grpsvcs Subsystem has been started. Subsystem PID is 17470. 0513-059 The grpglsm Subsystem has been started. Subsystem PID is 20824. 0513-059 The emsvcs Subsystem has been started. Subsystem PID is 19238.
| Q: | Are the statements outlined in bold normal? | |
Answers
| A: | Yes, especially after starting the first time. These services are started by HA on Node01, and by reboot on Node02. When stopping HA, it does not stop these particular services, so it is fine. |
Observation 5
When attempting to test the failover on the cluster; never be logged in as the maestro user. Since this user's home file system resides in the shared volume group (twsvg or /opt/tws), we will most likely have problems with:
-
The cluster actually failing over because it will not be able to mount the file system
-
Possible corruption of a file, or file system
Observation 6
The failover of the HACMP Cluster seems to work fine. We decided to benchmark the failover timings:
-
Shutdown of HA services on Node1 - Wed Oct 22 17:45:51 EDT 2003
-
Startup of HA services on Node2 Wed Oct 22 17:47:37 EDT 2003
Result: a failover benchmark of approximately 106 seconds.
The test is performed as follows. Have a machine that is external to the cluster prepared to ping emeamdm (9.149.248.77). This machine is called doswald.pok.ibm.com (you will need two terminals open to this machine).
-
In the first terminal, enter the UNIX "date" command (do not press Enter).
-
In the second terminal, enter the UNIX command ping 9.149.248.77 (do not press Enter).
-
Have terminals open to both nodes in the cluster. (We had both nodes in the cluster running the HACMP services, with the IBM Tivoli Workload Scheduler Resource Group running on Node1.)
Node1 must be in seen when selecting smit hacmp -> Cluster Services -> Stop Cluster Services -> "shutdown mode = takeover (press Enter only one time).
-
In the first terminal, from doswald, press Enter. This will give you the begin time of the cluster failover.
-
Very quickly go back to node1, and press Enter. This will start the cluster failover.
-
In the second terminal, from doswald, press Enter. This will execute the ping command.
-
In the first terminal, from doswald, enter the UNIX date command again (do not press Enter).
-
Wait for the ping command to resolve. Then press Enter for the final date command.
-
Subtract the first date command results from the second date command results.
| Q: | Does 106 seconds sound like a reasonable duration of time? | |
| Q: | Would the addition of another IP address possibly improve this failover time of 106 seconds? | |
| Q: | Would the addition of another IP address require another physical Ethernet card? | |
Answers
| A: | It does sound reasonable. However, the overall time should be the instant of failure until the time it takes for the application to get up and running by user connectivity on the other machine. You seem to only be testing IP connectivity time. You should also test via a hard failure, meaning halt the system. |
| A: | Only implementing IPAT via aliasing should improve this time (by perhaps a few seconds). |
| A: | No. |
|
| < Day Day Up > |
|