Contents

Dell Fluid Cache SAN 2.0 V2.0.0 Storage Deployment Guide For Linux Systems PDF

1 of 50
1 of 50

Summary of Content for Dell Fluid Cache SAN 2.0 V2.0.0 Storage Deployment Guide For Linux Systems PDF

Dell Fluid Cache for SAN Version 2.0.0 Deployment Guide for Linux Systems

Notes, cautions, and warnings NOTE: A NOTE indicates important information that helps you make better use of your computer.

CAUTION: A CAUTION indicates either potential damage to hardware or loss of data and tells you how to avoid the problem.

WARNING: A WARNING indicates a potential for property damage, personal injury, or death.

Copyright 2016 Dell Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. Dell and the Dell logo are trademarks of Dell Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.

2015 - 04

Rev. A03

Contents

1 Dell Fluid Cache for SAN Product Overview....................................................6 Key Features...........................................................................................................................................6

Connectivity...........................................................................................................................................6

2 Requirements for Fluid Cache............................................................................8

3 Fluid Cache Deployment Overview................................................................. 11

4 Downloading Firmware, Drivers, and Software............................................ 12

5 Preparing the Fluid Cache Components........................................................ 13 Checking Security Settings..................................................................................................................13

Checking Network Connections........................................................................................................ 13

RDMA Functionality....................................................................................................................... 14

Multipathing Support.....................................................................................................................14

Checking Application Settings............................................................................................................ 14

Preparing the Servers.......................................................................................................................... 14

Preparing the Cache Devices..............................................................................................................15

NVMe Cache Devices.................................................................................................................... 15

Non-NVMe Cache Devices........................................................................................................... 15

Preparing the Cache Network Adapters............................................................................................. 15

Bonding Network Adapter Ports................................................................................................... 16

Preparing the Cache Network Switch................................................................................................ 18

6 Installing and Setting up Fluid Cache............................................................. 19 Installing the Fluid Cache Software.................................................................................................... 19

Setting up the Fluid Cache Servers..................................................................................................... 19

7 Configuring and Managing Fluid Cache Clusters......................................... 21 Creating a Fluid Cache Cluster........................................................................................................... 21

Mapping Volumes in Fluid Cache.......................................................................................................22

Mapping Volume to Servers..........................................................................................................22

Mapping Volumes to a Subcluster................................................................................................23

Adding Servers to a Fluid Cache Cluster............................................................................................ 23

Adding Cache Devices to a Fluid Cache Cluster............................................................................... 24

Adding a Storage Center to a Fluid Cache Cluster............................................................................24

A Maintaining Fluid Cache Installations............................................................ 27 Removing Volume Mappings..............................................................................................................27

3

Removing Volume Mappings from a Server.................................................................................27

Removing Volume Mappings from a Subcluster......................................................................... 28

Removing Volumes.............................................................................................................................28

Removing a Server from a Fluid Cache Cluster.................................................................................28

Removing a Cache Device from a Fluid Cache Cluster.................................................................... 29

Deleting or Removing a Fluid Cache Cluster.....................................................................................29

Deleting a Cluster..........................................................................................................................29

Removing a Cluster.......................................................................................................................30

Removing the Fluid Cache Software............................................................................................30

Shutting Down and Restarting a Cluster............................................................................................30

Recreating a Fluid Cache Cluster........................................................................................................31

Uninstalling the Fluid Cache Software................................................................................................31

B Troubleshooting Fluid Cache Installations................................................... 33 Basic Troubleshooting Steps.............................................................................................................. 33

Cannot Set up Fluid Cache on a Server............................................................................................. 33

Incorrect Network Address or Netmask.............................................................................................34

Server Does Not Appear in List of Servers..........................................................................................34

Cache Device Does Not Appear in List of Cache Devices................................................................ 35

Cache Device Cannot Be Added to a Cluster....................................................................................35

Cannot Select a Specific Cache Mode ..............................................................................................35

Fluid Cache Node Is Unavailable in Enterprise Manager...................................................................36

Cached LUNs Are Unavailable............................................................................................................36

Cannot Create a Fluid Cache Cluster.................................................................................................36

Fluid Cache License Is Expired............................................................................................................37

Fluid Cache License Is Invalid............................................................................................................. 37

Cannot Configure the Cache Network.............................................................................................. 37

Cannot Assign or Remove a Storage Center..................................................................................... 39

Cannot Determine Which Cache Device Failed................................................................................ 39

Events for Fluid Cache Are Not Shown in Enterprise Manager.........................................................39

Cluster or Application Has Performance Issues................................................................................ 39

C Example Cabling Diagram................................................................................ 41 D Configuring a Dell Networking Switch..........................................................43 E Checking the System Configuration...............................................................45

Checking the Cache Device Firmware and Driver.............................................................................45

NVMe Cache Devices....................................................................................................................45

Non-NVMe Cache Devices...........................................................................................................45

Checking the Network Adapter Firmware and Driver....................................................................... 45

Checking iSCSI Status.........................................................................................................................46

Checking Storage Center Connectivity............................................................................................. 46

Checking Configured Servers.............................................................................................................46

Checking Linux Dependencies........................................................................................................... 47

Avahi Functionality........................................................................................................................ 47

4

F Related Documentation.................................................................................... 49 Accessing Enterprise Manager and Storage Center Documentation...............................................50

5

1 Dell Fluid Cache for SAN Product Overview Dell Fluid Cache for SAN is a server-side caching accelerator software. Fluid Cache makes high-speed PCI Express (PCIe) SSDs a shared, distributed cache resource. Fluid Cache is deployed on clusters of Dell PowerEdge systems connected using RoCE-enabled Ethernet adapters and operates within a SAN environment employing a Dell Compellent backing store.

Key Features

Fluid Cache supports the following features:

Write-back caching.

Accelerates reads, writes, and read-after writes.

Up to nine nodes in the cache cluster.

Dynamic cache expansion.

Centralized configuration, management, and reporting.

Connectivity

Fluid Cache operates on the following networks and interfaces:

Cache Network A high speed, low latency private network that Fluid Cache uses to connect the servers in the Fluid Cache cluster.

Management Interface

The connection to Dell Compellent Enterprise Manager, which manages Fluid Cache and the SAN.

Storage Area Network

The network that Dell Compellent Storage Center uses to handle data connectivity within the SAN.

6

Figure 1. Fluid Cache Connectivity

1. Fluid Cache nodes communicate with each other over a private network using RDMA.

2. Enterprise Manager creates, manages, and monitors the Fluid Cache clusters.

3. Fluid Cache nodes communicate with the Management IP (VIP) of the Storage Controllers.

4. Enterprise Manager manages the Dell Compellent array.

The interfaces illustrated in the figure are associated with specific ports (see Checking Network Connections). For a schematic of an example Fluid Cache installation, see Example Cabling Diagram.

7

2 Requirements for Fluid Cache

The following are prerequisites for deploying Fluid Cache for SAN. Refer to the Release Notes for the most recent requirements.

The supported versions of firmware, drivers, and software are required to run Fluid Cache. These are listed in the Release Notes and are available at dell.com/support. For download instructions, see Downloading Firmware, Drivers, and Software.

NOTE: For the current list of validated components, see the Dell Fluid Cache for SAN Compatibility Matrix available at http://www.dell.com/CacheSolutions.

Table 1. System Requirements

Servers The cluster on which you deploy Fluid Cache must contain a minimum of three server per each cache cluster and a maximum of nine servers.

Cache Devices There must be a cache device installed on at least two servers in the cluster.

Network Adapters A network adapter that supports Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE) must be installed on each server in the cluster.

Network Switch A compatible switch must be available for the cache network. A redundant switch configuration is recommended.

SAN Management Software

Enterprise Manager must be started and configured to manage the Storage Center in use with Fluid Cache.

SAN Connectivity All servers in the cluster must be connected to the SAN and be displayed on the Dell Compellent array.

Table 2. Hardware and Software Requirements

Element Requirement

Servers Dell PowerEdge servers that support Dell Express Flash PCIe SSDs:

Dell PowerEdge FC630 (FX2S chassis)

Dell PowerEdge M620

Dell PowerEdge M630

Dell PowerEdge M820

Dell PowerEdge R620

Dell PowerEdge R630

Dell PowerEdge T630

Dell PowerEdge R720

Dell PowerEdge R730

NOTE: Dell PowerEdge R730 servers support only SR-IOV and does not support Dell Express Flash PCIe SSDs.

8

Element Requirement

Dell PowerEdge R730XD

Dell PowerEdge R820

Dell PowerEdge R920

Dell PowerEdge T620

Operating Systems Red Hat Enterprise Linux (RHEL) 6.4 (64-bit)

Red Hat Enterprise Linux (RHEL) 6.5 (64-bit)

SUSE Linux Enterprise Server (SLES) 11 SP3 (64-bit)

Oracle Enterprise Linux (OEL) 6.5 (64bit)

Oracle Enterprise Linux (OEL ) 6.5 (64bit)

For Oracle Linux 6.4 (64-bit), use the RHEL 6.4 Fluid Cache RPM package.

NOTE: The Mellanox ConnectX-3 Driver, version 2.0.2.6.8 is not installed by default on OEL 6.4 systems. As a workaround, edit the Linux distribution (distro) file of RHEL6.4, and change the entry RHEL6.4 to OEL6.4.

RAM and hard disk drive space

10GB of available disk space

Minimum 32GB RAM, 64GB recommended

Cache Devices Dell Express Flash PCIe SSDs (175GB, 350GB, 400GB, 800GB, or 1.6TB)

Micron HHHL PCIe SSD cards (700GB or 1.4TB)

Network Adapters Mellanox ConnectX-3 Dual Port 10 GbE SFP+ Adapter

Mellanox ConnectX-3 Dual Port 40 GbE QSFP+ Adapter

Mellanox ConnectX-3 Dual Port 10 GbE KR Mezzanine Adapter

Fibre Channel HBAs QLogic 2560, single-port 8GB, Optical Fibre Channel HBA

QLogic 2562, dual-port 8GB, Optical Fibre Channel HBA

QLogic 2660, single-port 16GB, Fibre Channel HBA, full height

QLogic 2662, dual-port 16GB, Fibre Channel HBA, full height

Emulex LPE 12000, single-port 8GB, Fibre Channel HBA

Emulex LPE 12002, dual-port 8GB, Fibre Channel HBA

Emulex LPe16000B, single-port 16GB, Fibre Channel HBA

Emulex LPe16002B, dual-port 16GB, Fibre Channel HBA

Cache Network Switches

Supported switches include:

Dell Networking N4032F

Dell Networking N4064F

Dell Networking S4048-ON

Dell Networking S4810

Dell Networking S5000

Dell Networking S6000

Dell Networking Z9500

Dell Networking MXL Blade

Cisco Nexus 5548UP

9

Element Requirement

SAN Software Dell Compellent Enterprise Manager 2014 R1

Dell Compellent Enterprise Manager 2014 R2

Dell Compellent Enterprise Manager 2015 R1

Dell Compellent Enterprise Manager 2015 R2

Dell Compellent Storage Center SCOS 6.5.2

Dell Compellent Storage Center SCOS 6.6.5

Dell Compellent Storage Center SCOS 6.7.2

SAN Hardware Dell Compellent SC9000 controller

Dell Compellent SC8000 controller

Linux Dependencies During installation, RPM checks for the necessary Linux dependencies and alerts you to install any that are not found. Refer to the Release Notes for the complete list of required dependencies. To check which dependencies are currently installed on your system, run the command: rpm -qa

WARNING: Dell Fluid Cache for SAN requires Avahi traffic to run only on cache network interfaces running RoCE. Therefore, you must enable only the cache network running RoCE in the avahi-damon.conf file.

WARNING: Dell requires RoCE network dedicated only for Fluid Cache to avoid potential impact to traffic running on other interfaces including Oracle RAC traffic.

Optional: Fluid Cache Client Servers (do not have SSDs but participate in a cluster)

All Dell PowerEdge servers (or non-Dell servers with a supported operating system that can install a supported network adapter).

10

3 Fluid Cache Deployment Overview Complete the following tasks to deploy Fluid Cache for SAN. The sections referenced in each task give detailed instructions.

1. Download the required software, firmware, and drivers.

See Downloading Firmware, Drivers, and Software.

2. Identify and prepare the Fluid Cache servers and cache network switch.

See Preparing the Fluid Cache Servers and Cache Network Switch.

3. Install the Fluid Cache software on each server in the cluster.

See Installing and Setting Up Fluid Cache on Linux.

4. Create and configure the Fluid Cache cluster.

See Configuring and Managing Fluid Cache Clusters.

For more information about managing Compellent arrays, see the Dell Compellent Enterprise Manager Administrators Guide available at https://portal.compellent.com/. Also, see Accessing Dell Compellent Documentation.

11

4 Downloading Firmware, Drivers, and Software

NOTE: It is recommended that you update servers using iDRAC. For more information, go to en.community.dell.com/techcenter/b/techcenter/archive/2013/04/17/idrac7-now-supports- updating-server-components-using-racadm-and-web-gui.aspx.

1. Determine which firmware, drivers, and BIOS are installed for your system components by following the procedures in Checking System Configuration.

2. Go to dell.com/support.

3. Enter the Service Tag for a supported PowerEdge server and click Submit.

NOTE: Entering the Service Tag for any supported server allows you to download the correct firmware, drivers, and software for all supported servers.

4. Click the Get Drivers & Downloads tab.

5. Select an appropriate operating system.

Make sure that you select the correct operating system.

6. Download the necessary firmware, drivers, and software:

NOTE: More than one download may be available for the same component. Identify the latest software or firmware based on the release date.

a. To download BIOS, select BIOS.

If you have different models of servers in your Fluid Cache cluster, download the BIOS versions for each model.

b. To download cache device firmware and drivers, select Solid State Storage.

c. To download network adapter firmware, drivers, and the OFED package, select Network. Search for the Mellanox ConnectX-3 driver and firmware.

NOTE: Do not use a network adapter driver downloaded from the manufacturers site. Fluid Cache uses a custom driver provided only by Dell.

d. To download the Fluid Cache for SAN software, click Cache Solutions.

7. Download the files to an appropriate directory.

12

5 Preparing the Fluid Cache Components Before installing Fluid Cache, you must prepare the components of the Fluid Cache network: the servers, cache devices, network cards, and switches.

The instructions that follow assume that you have an existing SAN configured and managed. All nodes in the cache cluster must be connected to the SAN and visible on the Dell Compellent array. Instructions for racking and cabling a SAN solution are beyond the scope of this document. For more information about installing, upgrading, and configuring specific Dell components, see Related Documentation.

Checking Security Settings

Before you start the installation, make sure that:

Firewall settings do not block traffic or devices on the cache network, or on networks used by Fluid Cache (refer to the table of Port Specifications below).

SELinux, AppArmor, or other security frameworks do not limit applications or devices on the cache network.

iptables exceptions have been entered where appropriate. (Note that some default RHEL installations may create an iptables entry that blocks Fluid Cache. See the troubleshooting topic Cannot Configure Fluid Cache.

If your avahi-daemon.conf file does not use the default settings, make sure that Disable Publishing is set to no, Deny Interfaces does not list Fluid Cache interfaces, and Allow Interfaces is either commented out or lists all Fluid Cache interfaces.

NOTE: If any of these settings are incorrect, the Fluid Cache cluster cannot be configured.

Checking Network Connections

For Fluid Cache to function correctly, each Fluid Cache server must be able to communicate with other network components. Make sure that the following ports are available:

Table 3. Ports Specifications

Interface Port number Port type

Fluid Cache cluster Configuration Manager 7449 TCP

Node Auto-Discovery functionality 5353 UDP

Web-based server status tool 6773 and 6774 TCP (https)

(optional) Web-based server status tool 8080 and 8082 TCP (http)

Fluid Cache to Enterprise Manager 3033 TCP

13

Interface Port number Port type

Fluid Cache to Storage Center 3033 TCP

NOTE: Fluid Cache cannot be configured and does not function properly if any of these ports are unavailable. To diagnose network connection problems and reassign port numbers, see the troubleshooting topic Cannot Configure Fluid Cache.

RDMA Functionality

To make sure that RDMA is working correctly between the cluster nodes, use the utility ib_send_bw:

1. On one node, run the command: ib_send_bw 2. On a second node, run the command: ib_send_bw <IP_address_of_first_node>

If the test is successful, the bandwidth for the node displays on each node.

Multipathing Support

Fluid Cache supports multipathing and uses existing multipath configurations. You must make two modifications to the Devices section of the /etc/multipath.conf file on each node in the cluster:

1. For the vendor COMPELNT, specify no_path_retry 6 2. Add blacklist entries to exclude devices used by Fluid Cache, as shown:

blacklist { devnode "^fldc[0-9]*" devnode "^nvme[a-z]*" devnode "^rssd[a-z]*" }

Checking Application Settings All applications that use volumes mapped to Fluid Cache must be configured to start after Fluid Cache and exit before Fluid Cache.

Preparing the Servers

Make sure that each server in the Fluid Cache cluster has the latest supported BIOS version, Lifecycle Controller firmware, and iDRAC firmware.

For updating Dell Lifecycle Controller and BIOS firmware on 13th generation of PowerEdge servers, see the Dell Lifecycle Controller Graphical User Interface Version 2.05.05.05 For 13th Generation Dell PowerEdge Servers User's Guide, available at http://topics-cdn.dell.com/pdf/idrac8-with-lc- v2.05.05.05_User's%20Guide2_en-us.pdf

For updating Dell iDRAC firmware on 13th generation of PowerEdge servers, see the Integrated Dell Remote Access Controller 8 (iDRAC8) Version 2.05.05.05 User's Guide, available at http://topics- cdn.dell.com/pdf/idrac8-with-lc-v2.05.05.05_User's%20Guide_en-us.pdf

For other latest information related to Dell iDRAC and Dell Lifecycle Controller documentation, go to http://www.dell.com/support/home/us/en/19/product-support/product/idrac8-with-lc- v2.05.05.05/research

Modify your BIOS settings as follows:

14

1. Enter the system BIOS and set System Profile Settings to Performance.

2. Click Back, and then click OK to return to the main BIOS page.

3. Restart the server to activate the BIOS changes.

4. Repeat this process for each server in the Fluid Cache cluster.

Preparing the Cache Devices

Make sure that all cache devices used in the Fluid Cache cluster have the minimum firmware and driver versions specified in Requirements for Fluid Cache. To download updated firmware or drivers, see Downloading Firmware, Drivers, and Software.

Fluid Cache supports clusters containing cache devices of different sizes, but for the most predictable performance, it is recommended that the cache devices be of similar capacity.

NVMe Cache Devices

Firmware Refer to the Dell PowerEdge Express Flash NVMe PCIe SSD Users Guide for instructions about checking your current firmware version. Use the Dell Update Package to install the latest firmware.

Driver Refer to the Dell PowerEdge Express Flash NVMe PCIe SSD Users Guide for instructions about checking your current driver version and installing the updated driver.

Non-NVMe Cache Devices

Firmware Make sure the firmware is up to date by using this command: dmesg | grep mtip32xx | grep Firmware For each cache device, you must get results similar to the following, showing firmware version B1490508 or later:

mtip32xx 0000:46:00.0: Firmware Ver.: B1490508

NOTE: The firmware version must be B1490508 or later. Otherwise, even though they are visible in Enterprise Manager, the cache devices do not appear in the list of available devices when adding devices to a Fluid Cache cluster.

Driver Make sure the driver is up to date and running by using this command: dmesg | grep -i micron For each cache device, you must get results similar to the following, showing driver version 2.4.2 or later:

Micron RealSSD PCIe Block Driver Version 2.4.2

CAUTION: If the cache devices contain data, back up this data before adding the cache devices to the Fluid Cache cluster. All data on the cache devices are lost when they are added to the cluster.

NOTE: If your server already has cache devices installed, make sure that the cache devices you intend to use in the Fluid Cache cluster are not used by any other application.

Preparing the Cache Network Adapters 1. Install the network adapters in PCIe slots on the servers (x16 slots, if available).

2. Check the network adapter driver and firmware versions, and the OFED package version, and then update, if necessary (see Checking the Network Adapter Firmware and Driver).

15

NOTE: It is recommended to use an MTU of 1500 for the cache network.

NOTE: Do not use a network adapter driver downloaded from the manufacturers site. Fluid Cache uses a custom driver available only through Dell.

3. Create and configure an IP address on all RoCE network interfaces that are connected:

a. Open the interface configuration file by running the commands:

cd /etc/sysconfig/network-scripts vi ifcfg-

b. Edit the configuration parameters as given in the example here:

DEVICE=" " BOOTPROTO="static" ONBOOT="yes" IPADDR="<ip address of the network adapter>" NETMASK="<netmask of the cache network>" NAME=" "

c. Make a note of the address and netmask for later reference when configuring the cache network.

d. Save your changes to the configuration file.

e. Open the interface by running the command: ifup 4. If port bonding is used, refer to Bonding Network Adapter Ports.

5. If a blade enclosure is used, disable FlexAddress in the blade enclosure.

6. Repeat this process for each network adapter in the cache network.

NOTE: Make sure that all RoCE network adapters used by the Fluid Cache network are dedicated to the cache network and are not configured for any other network traffic.

Bonding Network Adapter Ports

Fluid Cache supports port bonding in active/passive mode (also called active/backup or master/slave).

Before implementing bonding on your network adapters, make sure that your cache network switches and cabling are correctly configured for bonding. For an example of a system cabled for port bonding, see Example Cabling Diagram.

NOTE: It is recommended that port bonding be used in configurations with redundant, uplinked cache network switches.

The following example describes configuring port bonding on an RHEL system using Dell Networking switches. For non-Dell Networking switches refer to the documentation specific to that networking switch.

1. Configure a virtual interface to be used as a bonded port (in this example, bond0) by running the following command:

vi /etc/modprobe.d/bond0.conf 2. In the configuration file, edit the parameters as follows:

alias netdev-bond0 bonding 3. Open the configuration file for the bonded port by running the following commands:

cd /etc/sysconfig/network-scripts

vi ifcfg-bond0 4. Edit the parameters in the configuration file as given in the example here:

DEVICE="bond0" BOOTPROTO="static"

16

ONBOOT="yes" IPADDR="<ip address of the network adapter>" NETMASK="<netmask of the cache network>" NAME="bond0" BONDING_OPTS="mode=1 fail_over_mac=1 miimon=100 downdelay=300 updelay=300"

NOTE: In the BONDING_OPTS line, the values for mode and fail_over_mac must be set

exactly as shown in the example.

5. Configure an interface for use on the bonded port by running the following command:

vi /etc/sysconfig/network-scripts/ifcfg-<interface name>

NOTE: For the names of the interfaces on your server used by Fluid Cache, run the command: ip addr. The names of the network interfaces used by Fluid Cache are often in the format

pXpY (for example, p6p2), but may be in the format ethX, emX, or other formats, depending on

your operating system. For example outputs, see Checking Network Adapter Firmware and Drivers.

6. In the configuration file, edit the parameters as follows:

DEVICE=<interface name> BOOTPROTO="none" ONBOOT="yes" NM_CONTROLLED="no" SLAVE="yes" MASTER="bond0"

7. For the other interface in the bonded port, repeat the tasks in 56.

8. Start network connection to the bonded port by running the following command: ifup bond0 9. Check the status of your bonded port and its interfaces by running the following command:

ifconfig The output must contain entries similar to the following. Note the line that states that the port session is started as a master or slave, and note that the slave interfaces no longer have individual IP addresses:

bond0 Link encap:Ethernet HWaddr 00:02:C9:E5:C5:C0 inet addr:172.20.3.116 Bcast:172.20.7.255 Mask:255.255.248.0 inet6 addr: fe80::202:c9ff:fee5:c5c0/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 . . . p5p1 Link encap:Ethernet HWaddr 00:02:C9:E5:C5:C0 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 . . . p5p2 Link encap:Ethernet HWaddr 00:02:C9:E5:C5:C0 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 . . .

10. Open the avahi-daemon.config file, and in the Server section, add this entry: Allow Interfaces = bond0.

11. Make sure that all servers in the cache network can ping every other server, and also the management interface.

17

Preparing the Cache Network Switch For a schematic of an example Fluid Cache installation, see Example Cabling Diagram.

For configuration tasks for a supported Dell Networking switch, see Configuring a Dell Networking Switch. For all other switches, see the manufacturers documentation.

For a list of supported switches, see Requirements for Fluid Cache.

1. Connect the ports on the network adapter to properly configured ports on a network switch.

NOTE: A redundant switch configuration is recommended.

If you have two switches for redundancy, on all of the cache network adapters, you must connect port 1 to one switch and port 2 to the other switch.

2. Make sure that all ports used in the cache network have the following settings:

The ports are in layer 2 mode.

The ports are in an untagged state.

The switch firmware is up to date.

Flow control (transmit and receive) is enabled and Data Center Bridging (DCB) is disabled.

NOTE: Enabling flow control is a requirement for Fluid Cache.

3. Save the running configuration.

4. To implement the changes, restart the switch.

18

6 Installing and Setting up Fluid Cache Before completing the tasks in this section, install the required Linux dependencies. See Checking Linux Dependencies.

Installing the Fluid Cache Software 1. Copy to the server the Fluid Cache tar.gz package that you downloaded earlier (see Downloading

Firmware, Drivers, and Software).

2. Expand the tar.gz package.

A new Fluid Cache directory is created, which contains an RPM file.

3. Change to the new Fluid Cache directory and run the following command:

rpm i <RPM file name>

There is a delay while the Fluid Cache software is installed on the server. During this process, RPM checks for the necessary Linux dependencies and asks you to install any that are not found. For a list of required dependencies, refer to the Release Notes available at the support site.

When processing is complete, the command prompt is displayed again.

NOTE: You can also install the software with a Yum repository using the following command: yum install <RPM file name>

4. Repeat tasks 13 for each server in the Fluid Cache cluster.

Setting up the Fluid Cache Servers Before setting up the servers in your Fluid Cache cluster:

Make sure that you have IP addresses configured for the network adapters on each server.

Make sure that you can ping the Mellanox interfaces on every server in the Fluid Cache cluster.

Record the following information for your servers (contact a network administrator, if necessary):

Management interface (used by Enterprise Manager to manage the SAN)

* Network address: ________________

* Netmask: ______________________

Cache network

* Network address: ________________

* Netmask: ______________________

19

An example of the required information is a device with an IP address of 172.18.1.2, whose network

address is 172.18.1.0, and netmask is 255.255.255.0.

1. Change to the following directory:/opt/dell/fluidcache/bin/ 2. Start the Host Cache Node (HCN) Setup tool by running the following command:

./hcn_setup.py

HCN Setup sets up a server for use as a Fluid Cache cluster node, and starts an agent on the server that allows it to be discovered by Enterprise Manager.

3. Press to begin the setup process.

A dialog box appears for a network address (for example, 172.18.1.0) and netmask (for example,

255.255.255.0) for the interface used by Enterprise Manager to manage the SANthe

management interface.

4. Enter the network address and press .

5. Enter the netmask and press .

6. When prompted to confirm your entries, press .

7. If you are using more than one management network, enter the address and subnet mask for the auxiliary network. Otherwise, press .

8. On the Cache Network page, enter the network address and press .

9. Enter the netmask and press .

10. When prompted to confirm your entries, press .

A message is displayed indicating that configuration was successful and prompting you to start Fluid Cache.

11. Press to start the Fluid Cache software.

A message is displayed indicating that the server is configured for Fluid Cache.

12. Repeat this process for the other servers in the Fluid Cache cluster.

You must set up Fluid Cache on at least three servers before creating a Fluid Cache cluster.

NOTE: If configuration is unsuccessful because of an incorrect value for a network address or netmask, restart HCN Setup and enter the correct values.

After setting up the Fluid Cache servers, create and configure Fluid Cache clusters in Enterprise Manager. See Configuring and Managing Fluid Cache Clusters.

NOTE: HCN Setup also supports a CLI unattended installation. The commands used are:

hcn_setup.py -i <management interface IP address>:<subnet mask> -I <cache network IP address>:<subnet mask> -u [for unattended installations] -r [to run agent]

20

7 Configuring and Managing Fluid Cache Clusters Make sure that your system meets the following prerequisites:

The Fluid Cache software must be installed and running on all servers to be used in the Fluid Cache cluster.

All servers in the Fluid Cache cluster must be connected to the SAN and be visible on the Compellent array.

Enterprise Manager must be running and configured to manage the Storage Center to be used with Fluid Cache.

Your Enterprise Manager user account must have Administrator or Volume Manager privileges.

The Fluid Cache license file must be stored on the Enterprise Manager server used to create the Fluid Cache cluster or on a shared folder available to it.

Creating a Fluid Cache Cluster 1. Log in to Enterprise Manager as a user with Administrator or Volume Manager privileges.

2. In Enterprise Managers Storage view, expand Storage Centers if necessary, and then select a Storage Center.

3. In the Storage pane, select Dell Compellent.

4. In the Summary tab, click Configure Fluid Cache Cluster to launch the configuration wizard. (Do not click Add FluidFS Cluster.)

The Discover Fluid Cache Servers page of the configuration wizard is displayed.

5. Type or select appropriate data in the Discover Fluid Cache Servers window.

a. In the Host or IP Address box, type the host name or IP address associated with the management network of any available Fluid Cache server.

b. The Port box is autopopulated. Change only if necessary.

c. In the User Name box, type the username , which is fldc.

You can also use the root user name and password in these boxes, if available.

d. In the User Password box, type the password . The default value is calvin.

NOTE: If you change these default properties on any node in a Fluid Cache cluster, you must change them on all nodes. The login and password must be the same on all nodes in the Fluid Cache cluster.

e. Click Next.

The Select Servers window is displayed.

6. By default, all available servers are selected. Clear the check box next to unwanted servers or select the Unselect All option, and then select three or more servers to be included. (Click Select All to use all available servers again.)

7. Click Next.

21

The Cluster Settings page of the configuration wizard is displayed.

8. Type or select appropriate data in the Cluster Settings window.

a. In the Name box, enter a name for the cluster.

b. Click Browse next to the License File box.

The Select Fluid Cache License File dialog box is displayed.

c. Browse to the location of the license file, select the file, and then click Save.

d. Verify that the license file and path displayed are correct and click Next.

The system processes for a few minutes while the cluster is created and the system automatically identifies compatible servers and devices on the network. After this process is completed, the Select Devices window is displayed.

9. By default, all available Fluid Cache devices are selected. Clear the check box next to unwanted devices or select the Unselect All option, and then select the required devices. (Select the Select All option to use all available devices again.)

CAUTION: You will lose any existing data on the cache devices when they are added to the Fluid Cache cluster. Back up this data before proceeding.

10. Click Next.

The Select Storage Centers page of the configuration wizard is displayed.

11. In the Select Storage Centers page, select one or more Storage Centers to include in the Fluid Cache cluster, and then click Finish.

After a delay while the system processes, the Storage pane contains a new top-level folder named Fluid Cache Clusters. Inside this folder is the Fluid Cache cluster that was just created. Fluid Cache clusters are denoted by a blue circle with the letters FC.

Select the Fluid Cache cluster. In the Summary tab, different areas of the window show the cluster's status, servers, devices, and other information.

After creating a Fluid Cache cluster, map volumes to the cluster. See Mapping Volumes.

Mapping Volumes in Fluid Cache

In Enterprise Manager, volume mappings created for a server in a Fluid Cache cluster behave similar to any other volume mappings. Volumes can be mapped either to an individual server, or to a server cluster (a subcluster) within the Fluid Cache cluster.

NOTE: If you have a shared data application, such as a clustered file system or clustered application, map volumes to a subcluster.

NOTE: Any applications that were using a volume before it was cached must be modified to use the new cached volume. When a cached volume is mapped to one or more servers, Fluid Cache creates its own Linux device file, /dev/fldcX. (The first device mapped is /dev/fldc0, the second

is /dev/fldc1, and so on.) Any application that uses the cached volume must be configured to

use the new Fluid Cache device rather than the /dev/sdX or /dev/mpathX device. If the cached

volume is shared by multiple servers, the /dev/fldcX device is available on each server.

Mapping Volume to Servers

Before mapping a volume to a server node, create a Fluid Cache cluster (see Creating a Fluid Cache Cluster).

22

To map a volume to a server:

1. In the Storage view, expand Storage Centers if necessary, and then select the Storage Center that contains the appropriate volume.

2. In the Storage tab, expand Volumes if necessary, and then locate the volume you want to map.

3. Right-click the volume and select Map Volume to Server.

4. In the Map Volume to Server window, select the server.

5. Click Next.

6. Select Enable Fluid Cache.

7. From the Host Cache Policy drop-down menu, select a cache mode:

Write-back (default): In addition to caching reads, write-back mode allows the caching of written data without waiting for the Compellent Array to acknowledge the write operation. Write-back caching requires a cache device on two or more servers in the cluster.

Write-through: Write-through mode forces writes to both the cache and the Compellent Array simultaneously. Warm reads and read-after writes are accelerated but write operations are not. Write-through caching requires only one cache device on one server in the cluster.

NOTE: The cache mode selected for a volume mapping cannot be changed. To select a different cache mode, you must remove the mapping and create a new mapping. When you create the new mapping, you can then select a different cache mode.

8. (Optional) Select the Keep cached data on the node that accessed the data option. Selecting this option gives the best performance for warm reads from the cache for datasets that fit on the local nodes cache devices, because all data is local to the client. If this option is not selected, cached data is evenly distributed among all cache devices.

9. Click Finish.

At this stage, there is a delay while the system makes a number of configuration changes.

Mapping Volumes to a Subcluster

Before mapping volumes to a subcluster, create a Fluid Cache cluster (see Creating a Fluid Cache Cluster) and a server cluster (subcluster) within it. To map a volume to a subcluster, follow the procedure for mapping a volume to a server (see previous section), but instead of selecting a server in the Map Volume to Server window (step 4), select a subcluster within a Fluid Cache cluster.

Although not specific to Fluid Cache, be aware of these aspects of volume mappings created at the subcluster level in Enterprise Manager:

When servers are added to or removed from the subcluster, they automatically inherit or disinherit the subclusters volume mappings.

You can promote a volume mapping from a server to the subcluster, and demote it from the cluster to the subcluster.

If a server has existing mappings, the server keeps those mappings when it becomes part of a subcluster.

Adding Servers to a Fluid Cache Cluster To add servers to a Fluid Cache cluster:

1. Log in to Enterprise Manager as a user with Administrator or Volume Manager privileges.

2. In the Storage view, expand Fluid Cache Clusters if necessary.

23

3. Right-click the Fluid Cache cluster (not Fluid Cache Clusters itself) and click Tasks Add Servers to Cluster.

The Add Servers to Cluster dialog box is displayed. By default, all servers are selected that have Fluid Cache installed.

4. Clear the check boxes next to unwanted servers or select the Unselect All option, and then select the servers to be added. (To use all available servers again, select the Select All option.)

5. Click OK.

The system processes for some time, and then the server appears inside Fluid Cache Clusters in the Storage tab.

The Add Devices to Cluster dialog box is displayed. By default, all devices compatible with Fluid Cache are selected.

6. Clear the option next to unwanted cache devices or click the Unselect All option, and then select the cache devices to be added. (Select the Select All option to use all available cache devices again.)

CAUTION: Any existing data on a cache device is lost when the device is added to the Fluid Cache cluster. Back up this data before proceeding.

7. Click OK.

The device now appears in the list in the Devices section.

If you have a shared data application such as a cluster file system or clustered application, you may want to add the server to a server cluster (a subcluster) inside the Fluid Cache cluster. See Mapping Volumes. To create a subcluster, refer to the Enterprise Manager Administrators Guide.

Adding Cache Devices to a Fluid Cache Cluster 1. Log in to Enterprise Manager as a user with Administrator or Volume Manager privileges.

2. In the Storage view, expand Fluid Cache Clusters if necessary, and then right-click the Fluid Cache cluster. (Do not right-click Fluid Cache Clusters itself.)

3. Click Tasks Add Devices to Cluster.

The Add Devices to Cluster window is displayed. By default, all available devices compatible with Fluid Cache are selected.

4. Clear the option next to unwanted cache devices or click the Unselect All option, and then select the cache devices to be added. (Click the Select All option to use all available cache devices again.)

CAUTION: Any existing data on a cache device is lost when the device is added to the Fluid Cache cluster. Back up this data before proceeding.

5. Click OK.

The devices now appear in the list in the Devices section.

Adding a Storage Center to a Fluid Cache Cluster 1. Log in to Enterprise Manager as a user with Administrator or Volume Manager privileges.

2. Click the Storage view.

3. In the Storage pane, expand Fluid Cache Clusters if necessary, and then select the Fluid Cache cluster.

4. In the Cache tab, select Tasks and click Assign Storage Centers.

The Assign Storage Centers window is displayed.

5. In the Assign Storage Centers window, select one or more Storage Centers to be added.

24

6. Click OK.

25

26

A Maintaining Fluid Cache Installations This chapter describes tasks that must occasionally be performed after initial Fluid Cache deployment.

WARNING: When performing any operation to a server platform that hosts Fluid Cache software, you must perform either one of the following methods:

Shut down a single Fluid Cache Cluster node at a time and perform the host-based server updates.

After the updates to host server are complete, reboot the host and bring the Fluid Cache Cluster node online. Before shutting down and updating the next Fluid Cache cluster node, verify that the Fluid Cache Cluster node is added to Fluid Cache Cluster appropriately.

Place the entire Fluid Cache Cluster into maintenance mode, and then shut down the entire Fluid Cache Cluster by shutting down the Fluid Cache Cluster nodes one after the other.

After the updates to the host servers are complete, reboot the server and bring the Fluid Cache Cluster nodes online.

WARNING: To avoid potential performance issues and data loss, you must NOT shutdown multiple Fluid Cache Cluster nodes at the same time to perform Fluid Cache host server based updates.

Removing Volume Mappings

To see a list of Fluid Cache mappings, select the appropriate Storage Center in the Storage view, and in the in the Storage tab, select the Fluid Cache cluster. The Fluid Cache mappings for the cluster are listed in the Volumes area at the bottom of the page.

Removing Volume Mappings from a Server

1. Make sure the volume is no longer in use.

2. In Enterprise Managers Storage view, expand Storage Centers if necessary and select the appropriate Storage Center. (Do not select Fluid Cache Clusters or its contents.)

3. In the Storage tab, expand Servers if necessary and locate the server whose Fluid Cache mappings you want to remove.

4. Right-click the server and select Remove Mappings.

The Remove Mappings window is displayed.

5. In the Remove Mappings window, select the volume and click OK.

There may be some delay while the mapping is removed and dirty data in the cache is flushed to main storage. When the mapping removal is complete, the volumes icon in the Storage tab turns from blue to gray color. You may need to manually refresh the display.

27

NOTE: The Fluid Cache mappings for the server are listed in the Volumes area at the bottom of the page.

Removing Volume Mappings from a Subcluster

1. In Enterprise Managers Storage view, expand Storage Centers if necessary and select the appropriate Storage Center. (Do not select Fluid Cache Clusters or its contents.)

2. In the Storage tab, expand Servers, and then the Fluid Cache clusters if necessary and select the subcluster whose mappings you want to remove.

3. In the right pane, below the list of servers, select the volume and click Remove Mappings.

There may be some delay while the mapping is removed and dirty data in the cache is flushed to main storage. When the mapping removal is complete, the volumes icon in the Storage tab turns from blue to gray color. You may need to manually refresh the display.

NOTE: The Fluid Cache mappings for the subcluster are listed in the Volumes area at the bottom of the page.

NOTE: All Fluid Cache volume mappings must be removed before a server can be removed from a Fluid Cache cluster.

Removing Volumes

To avoid the potential for performance issues and data loss, it is a recommended best practice to unmount, remove, and delete all volume information after the volume is no longer in use.

Note that this is a general guideline pertaining to all Dell Compellent installations, and is not specific to Fluid Cache.

For more information and the procedure for removing volume information, refer to the section about removing volumes in Red Hat Enterprise Linux (RHEL) 6x Best Practices.

Removing a Server from a Fluid Cache Cluster 1. If the server belongs to a server cluster (a subcluster) within a Fluid Cache cluster, remove the

server from the subcluster:

a. Prior to removing the server from a Fluid Cache cluster, you must shutdown the host or stop the Fluid Cache service.

b. In Enterprise Managers Storage view, select the appropriate Storage Center. (Do not select Fluid Cache Clusters or its contents.)

c. In the Storage tab, expand Servers if necessary and locate the server.

d. Right-click the server and select Remove Server from Cluster. e. When asked to confirm the action, click OK.

In the Storage tab, the server now appears outside of the subcluster, but is still inside the Fluid Cache cluster.

NOTE: Removing the server from the subcluster removes all mappings created for the subcluster.

2. Remove all Fluid Cache mappings from all volumes mapped to that server (see Removing Volume Mappings). Note that you do not have to remove non-Fluid Cache mappings.

3. Remove the server from the Fluid Cache cluster:

a. In the Storage view, select the Fluid Cache cluster. (Do not select Storage Center or its contents.)

28

b. In the Summary tab, locate the server in the Servers section.

c. Right-click the server and select Remove Server from Cluster. d. When asked to confirm the action, click OK.

After the system processes for some time, the server reappears outside of the cluster in the Servers area. You may need to manually refresh the display.

NOTE: If you exit Fluid Cache on a server and then remove the server from a cluster, you must run HCN Setup on the server to add the server to any cluster, including the cluster from which it was removed.

NOTE: To completely remove the Fluid Cache software from the server, run this command: rpm -e $(rpm -qa | grep FLDC). Although this command removes all configuration

information, in some cases you may also want to remove all components of Fluid Cache from your installation by deleting the installation directory: /opt/dell/fluidcache/ NOTE: Do not disable the Fluid Cache agent service running the chkconfig command.

Removing a Cache Device from a Fluid Cache Cluster If you are removing the cache device from the server, perform a graceful removal by first ensuring that the cache device is not in active use. See Removing a Server from a Fluid Cache Cluster for the steps on gracefully removing a server from a cluster.

1. In Enterprise Managers Storage view, expand Fluid Cache Clusters if necessary, and then select the Fluid Cache cluster. (Do not select Storage Centers or its contents.)

2. In the Summary tab, locate the cache device in the Devices section.

3. Right-click the device and select Remove Device from Cluster.

4. When asked to confirm the action, click OK.

As part of the deletion process, dirty data in the cache is flushed to main storage. This could take a considerable duration of time for a large quantity of data.

When processing is complete, the cache device no longer appears in the list of devices. You may need to manually refresh the data on the page.

Deleting or Removing a Fluid Cache Cluster There are important differences between deleting and removing a Fluid Cache cluster:

Deleting a Fluid Cache cluster deletes the cluster and all of its configuration information from Enterprise Manager, deletes the Storage Center object, and resets the configuration data on the Fluid Cache nodes so that they can be added to another cluster.

Removing a cluster removes it from Enterprise Manager but the nodes still contain configuration data for the cluster, which must be removed before the nodes can be added to another cluster.

NOTE: Deleting a cluster is the preferred action. Remove a cluster only if deleting it is not possible because the cluster is not functioning normally.

Deleting a Cluster

1. Stop all I/O on cached volumes.

2. Unmount cached volumes.

29

3. Remove all Fluid Cache mappings from all volumes in the cluster (see Removing Volume Mappings). Note that you do not have to remove non-Fluid Cache mappings.

4. In Enterprise Managers Storage view, right-click the Fluid Cache cluster and click Delete.

5. When prompted to confirm the deletion, click OK.

There may be a delay while the deletion is processed.

6. Shut down the Fluid Cache hosts.

Removing a Cluster

1. Stop all I/O on cached volumes.

2. Unmount cached volumes.

3. In Enterprise Managers Storage view, right-click the Fluid Cache cluster and click Remove.

4. In Enterprise Managers left pane, select the Storage Center for the cached volume.

5. In the Storage tab, expand Servers.

6. If the cluster still has the blue circle with the letters FC to indicate that it is a Fluid Cache cluster, wait a minute or two and refresh the view.

7. If the blue FC circle persists, power down the hosts and refresh the view.

8. After the blue circle is replaced with a red X, right-click the cluster and click Delete.

9. Shut down the Fluid Cache hosts.

Removing the Fluid Cache Software

NOTE: To completely remove the Fluid Cache software from the server, run the command: rpm -e $(rpm qa | grep FLDC). Although this command removes all configuration information, in some cases you may want to remove all components of Fluid Cache from your installation by also deleting the installation directory: /opt/dell/fluidcache/

Shutting Down and Restarting a Cluster Shut down a cluster if, for example, you need to perform system maintenance but do not need to make any configuration changes to the cache network itself.

To shut down a cluster:

1. Exit any applications that access cached volumes.

2. In Enterprise Manager, click Storage.

3. In the Storage pane, expand Fluid Cache Clusters if necessary, and then select the Fluid Cache cluster.

4. Click Shutdown.

The Shutdown window is displayed.

5. Click Yes.

The system processes for some time while data is flushed. There may be a significant delay for large amounts of data.

NOTE: While a cluster is shut down, all cached volumes and their data are inaccessible. If you need to maintain access to these volumes while the cluster is shut down, remap the volumes before shutting down the cluster.

To restart a cluster:

30

1. Make sure the SAN and Storage Center are fully operational, and the cache network is configured and running.

2. In Enterprise Manager, click the Storage view.

3. In the Storage pane, expand Fluid Cache Clusters if necessary and select the Fluid Cache cluster.

4. Click Restart.

Recreating a Fluid Cache Cluster Although it is not part of standard operations or maintenance, in some instances you may want to recreate a Fluid Cache cluster.

1. Make sure that no cached LUNs on the servers are in use.

NOTE: All cached data on a server is lost when you perform the following procedure. Flush this data before proceeding.

2. Run the Host Cache Node Setup tool on each node in the cluster. See Setting up a Fluid Cache Node. This removes all existing configuration information and settings.

3. Create and configure a new cluster. See Configuring and Managing Fluid Cache Clusters.

Uninstalling the Fluid Cache Software Uninstall the Fluid Cache software if a server will no longer be used for Fluid Cache.

Before uninstalling the software, you must remove all volume mappings and remove the server from its Fluid Cache cluster. See Removing Volume Mappings and Removing a Server from a Fluid Cache Cluster.

To uninstall the Fluid Cache software, run the command:

rpm -e $(rpm qa | grep FLDC)

Although this command removes all configuration information, in some cases you may want to remove all components of Fluid Cache from your installation by also deleting the installation directory: /opt/ dell/fluidcache/

31

32

B Troubleshooting Fluid Cache Installations If you have issues running Fluid Cache after a successful completion of the installation procedure, contact your Compellent Copilot.

Troubleshooting the Compellent array and SAN architecture is beyond the scope of this document.

For additional troubleshooting information, refer to the Enterprise Manager Administrators Guide and the documentation for other hardware and software components. See Related Documentation.

After the initial list of Basic Troubleshooting Steps, the troubleshooting topics are presented in the order in which the issues are likely to appear as you deploy, configure, administer, and maintain a Fluid Cache installation.

Basic Troubleshooting Steps

Make sure the following conditions are fulfilled:

You have downloaded and installed the most recent firmware, drivers, and software required to support Fluid Cache for SAN (see Requirements for Fluid Cache).

The hardware is racked and cabled according to your hardware documentation.

The network security settings match those specified in Checking Network Connections.

The ports available to Fluid Cache match those specified in Connectivity.

All servers are in the same management interface subnet and in the same cache network subnet.

Each server in the Fluid Cache cluster appears in Enterprise Managers list of servers.

The avahi-daemon is running (see Checking System Configuration).

(Optionalfor systems with multipath enabled and running) The Fluid Cache devices have been excluded in the /etc/multipath.conf file on each server in the cluster. See Checking Network Connections.

Cannot Set up Fluid Cache on a Server

Possible Cause HCN Setup could not set up Fluid Cache on a server because one of the Linux dependencies is not installed.

Solution Follow the instructions given by RPM, which should state the dependencies that are missing. Check for the dependencies listed in the Release Notes and compare them with the list of currently installed dependencies, which you can check by running this command: rpm -qa Fluid Cache uses Avahi for autodiscovery between cluster nodes. To check that the Avahi daemon is running on a node, run the command: ps ef | grep avahi- daemon. To start the Avahi daemon, run the command: /etc/init.d/avahi-

33

daemon start. To ensure that the Avahi daemon starts when the server is

restarted, run the command: chkconfig avahi-daemon on. If your avahi- daemon.conf file does not use the default settings, make sure that Disable Publishing is set to no, that Deny Interfaces does not list Fluid Cache interfaces,

and that Allow Interfaces is either commented out or lists all Fluid Cache interfaces.

Possible Cause HCN Setup could not set up Fluid Cache on the server because the MPIO service was configured for multipathing, but devices required by Fluid Cache were not blacklisted in the etc/multipath.conf file.

Solution Add blacklist entries for the devices required by Fluid Cache. On each node in the cluster, modify the Devices section of the /etc/multipath.conf file as follows:

blacklist { devnode "^fldc[0-9]*" devnode "^nvme[a-z]*" devnode "^rssd[a-z]*" }

After creating the entries and saving the file, start Fluid Cache by running this command: service fldc_agentd start

Incorrect Network Address or Netmask

Possible Cause While setting up a server, incorrect values were entered for the network address or netmask of either the management interface or the cache network.

Solution Restart HCN Setup and enter the correct values. See Setting up the Fluid Cache Servers.

Server Does Not Appear in List of Servers

Possible Cause A configuration issue is preventing the server from appearing in the list.

Solution From the server, run the command ip addr. The cache network interfaces state

should display as UP. If not, recheck the server configuration. See Preparing the

Fluid Cache Servers and Setting up the Fluid Cache Servers.

Possible Cause Firewall or iptables settings are preventing network communication.

Solution Check your firewall and iptables settings. See Checking Security Settings.

Possible Cause The network switch is not correctly cabled or configured.

Solution Review the settings for the network switch and consult your switch documentation. See Cluster or Application Has Performance Issues.

Possible Cause The Fluid Cache agent is not running on the server.

Solution From the server, run the command: service fldc_agentd start Possible Cause The Avahi daemon is not running on the server.

34

Solution From the server, run the command: service avahi-daemon start. See also

Checking Linux Dependencies.

Cache Device Does Not Appear in List of Cache Devices

Possible Cause The cache device you are trying to add is not supported by Fluid Cache.

Solution Refer to the list of supported devices in Requirements for Fluid Cache.

Possible Cause The driver or firmware for the cache device is not a supported version.

Solution Make sure that your cache device firmware and driver are up to date and that the driver is loaded.

Firmware: Run the command: dmesg | grep mtip32xx | grep Firmware

In the output, the firmware must version displayed must be B1490508 or later.

Driver: Run the command: dmesg | grep -i micron

In the output, the driver version for the device named Micron RealSSD PCIe must be 2.4.2 or later.

Possible Cause The cache device is not functioning properly.

Solution To check device function, select the device in Enterprise Manager and in the Event tab, look for a device failure message. Replace the cache device if necessary, using instructions in the Dell Compellent Enterprise Manager Users Guide.

Cache Device Cannot Be Added to a Cluster

Possible Cause The cache device is not functioning properly. Under some conditions, the process of adding a device completes normally even though the device being added is not functioning properly.

Solution To check device function, select the device in Enterprise Manager and in the Event tab, look for a device failure message.

If necessary, replace the cache device by following the instructions provided in the Dell Compellent Enterprise Manager Users Guide.

Cannot Select a Specific Cache Mode

Possible Cause There is an existing cache mode configured for the volume mapping. The cache mode chosen for a volume mapping cannot be changed. The existing mapping must be deleted, and a new mapping created. A new cache mode can be selected while creating the new mapping.

35

Solution Remove the volume mapping and create a new mapping. When you create the new mapping, you can then select a different cache mode (cache policy). See Removing Volume Mappings and Mapping Volumes.

Possible Cause The cluster is in maintenance mode.

Solution Take the Fluid Cache cluster out of maintenance mode by selecting the cluster in Enterprise Manager, clicking Edit Settings in the Summary tab, and clearing the Maintenance Mode option. Note that certain system failures or an invalid license may prevent the cluster from being taken out of maintenance mode. Refer to the Enterprise Manager Status and Events tabs, and see Fluid Cache License Is Expired.

Fluid Cache Node Is Unavailable in Enterprise Manager

Possible Cause The node failed during restart. The node was unable to perform a graceful shutdown because the chkconfig command was used to shut down the Fluid

Cache agent service.

Solution Do not use the chkconfig command to disable the Fluid Cache agent service.

Perform normal node recovery procedures to return the node to normal operation.

Cached LUNs Are Unavailable

Possible Cause After restarting Fluid Cache or restarting a server, cached LUNs may not be immediately available.

Solution Wait for the cached LUNs to reappear. It may take a significant amount of time to write a large amount of data. The cached volumes reappear after the cache completes recovery.

Possible Cause A fault in the cluster is preventing the volume from being recognized by Enterprise Manager.

Solution Check the Enterprise Manager status page and event log for error messages.

Cannot Create a Fluid Cache Cluster

Possible Cause A minimum of three PowerEdge servers have not been configured for use with Fluid Cache. Until at least three servers have Fluid Cache installed and configured, the option to create a Fluid Cache cluster is unavailable.

Solution Add three or more nodes and configure the nodes before attempting to create a Fluid Cache cluster in Enterprise Manager.

Possible Cause Network connectivity issues are preventing the creation of a Fluid Cache cluster.

Solution Check that the security settings are correct and the required ports and network connections are open. See Checking Network Connections and Checking Security Settings.

36

Fluid Cache License Is Expired

Possible Cause System settings such as changes to the system date cause the current Fluid Cache license to expire. You can still access data on cached volumes, but performance is degraded because the Fluid Cache cluster has been placed in maintenance mode and caching is no longer active.

Solution Check the status of the license file by selecting the Fluid Cache cluster in Enterprise Manager and referring to the status shown on the Events or Cache tabs. If the license is expired, make sure that your system settings for date and time are correct. If it is not correct, set the appropriate date and time. Take the Fluid Cache cluster out of maintenance mode by selecting the cluster in Enterprise Manager, clicking Edit Settings in the Summary tab, and then clearing the Maintenance Mode option.

Possible Cause Fluid Cache for SAN is running on an evaluation license (typically 90 days) and that time period has been exceeded. You can still access data on cached volumes, but performance is degraded because the Fluid Cache cluster has been placed in maintenance mode and caching is no longer active.

Solution Check the status of the license file by selecting the Fluid Cache cluster in Enterprise Manager and referring to the license type shown on the Events or Cache tabs. If the number of days remaining is zero, contact your Dell representative to purchase a Fluid Cache for SAN license. After activating the new license, take the Fluid Cache cluster out of maintenance mode by selecting the cluster in Enterprise Manager, clicking Edit Settings in the Summary tab, and deselecting Maintenance Mode.

Fluid Cache License Is Invalid

Possible Cause The license file is invalid if it is in any way modified. This causes unsuccessful digital signature validation.

Solution Contact Dell Customer Support.

Cannot Configure the Cache Network

Possible Cause Firewall settings or IP table entries are preventing access to one or more ports required by Fluid Cache.

Solution Change your firewall settings to allow access by Fluid Cache. For a list of required ports, see Checking Network Connections. To check which ports are currently in use, log in as a root user and run the command:

/bin/netstat -tulpn

Also, check for iptables entries that may be blocking Fluid Cache network traffic. Note that some default installations for RHEL create an iptables entry for

37

ib_send_bw that prevents connections to another server and thus blocks Fluid

Cache network traffic.

Possible Cause One of the ports required by Fluid Cache is in use by another process.

Solution Refer to the required ports listed in Checking Network Connections and reassign ports as needed.

Check which ports are in use by logging in as a root user and running this command:

/bin/netstat -tulpn To change the ports used by Fluid Cache, open the /etc/services file on

each server and add the following information, modifying the port numbers as needed:

fldc-http 8080/tcp # Fluid Cache CFM Web Server fldc-https 6773/tcp # Fluid Cache CFM Web Server fldca-http 8082/tcp # Fluid Cache Agent Web Server fldca-https 6774/tcp # Fluid Cache Agent Web Server The existing entry for the mDNS service on UDP port 5353 may also need to be modified. It is used by the Avahi daemon, which is required by Fluid Cache.

After changing the ports, restart the Fluid Cache agent service by running the following command:

service fldc_agentd restart

NOTE: Port 6774 is used to communicate with Enterprise Manager. If you change the port number for fldca-https in the etc/services file, you

must also configure Enterprise Manager to use the new port number.

NOTE: Port 3033 is used to communicate with Enterprise Manager and Storage Center. To configure those applications to use a different port, refer to the corresponding Administrators Guide. After a new port is configured in Enterprise Manager or Storage Center, Fluid Cache automatically uses the new port.

NOTE: Port 7449 is required by Fluid Cache. If another application is using this port, configure that application to use a different port.

Possible Cause The network adapters are not configured for Ethernet functionality, and so the ethtool and ifconfig commands cannot be used to configure the network

adapters.

Solution On each node in the Fluid Cache cluster, run the command: connectx_port_config In the configuration file, if the entry for ConnectX PCI Devices references auto or ib, change the setting for Select mode for port x to 2.

38

Cannot Assign or Remove a Storage Center

Possible Cause The Storage Center is already assigned to another Fluid Cache cluster.

Solution In Enterprise Manager, see whether or not a Storage Center is listed for the Fluid Cache cluster.

Possible Cause Network connectivity issues are preventing Enterprise Manager from communicating with Storage Center.

Solution Make sure the network is functioning properly. Refer to Checking Network Connections, Checking Security Settings, and Checking Storage Center Connectivity.

Cannot Determine Which Cache Device Failed

Possible Cause One of the cache devices has failed on a server with multiple cache devices installed, and it is not clear which of them failed.

Solution In Enterprise Manager, in the Cache tab, the failed cache device is identified by a red X through it. Note the last digits of the number for this cache device. This number matches the serial number printed on the label of the failed cache device.

Events for Fluid Cache Are Not Shown in Enterprise Manager

Possible Cause When Enterprise Manager was installed, the Data Collector was not configured for the correct IP address.

Solution Make sure the IP address that the Data Collector uses is accessible by the Fluid Cache nodes. To view the IP address, start the Data Collector Manager, select General Information, and note the IP address within the URL in the Web Site field. Change the address if necessary and restart the Data Collector.

Cluster or Application Has Performance Issues

Possible Cause One or more cache devices are uninstalled, have failed, or do not have the correct firmware or drivers.

Solution Use Enterprise Manager to check the functionality of the cache devices. See also Checking the Cache Device Firmware and Driver.

Possible Cause The Compellent storage array is overloaded.

Solution In Enterprise Manager, check the storage latencies and throughput on the cached volumes. Add more capacity to the Compellent array if necessary.

Possible Cause The application is not making use of the cache.

39

Solution Make sure the application has an I/O profile that can leverage Fluid Cache, which accelerates reads, writes, and read-after writes. If the application does, check the running cache mode for that volume in Enterprise Manager. If necessary, change cache modes by deleting the volume mapping and creating a new one with a different cache mode. See Mapping Volumes.

Possible Cause The active data set greatly exceeds the size of the cache pool.

Solution Add more cache devices to increase the size of the cache pool.

Possible Cause Network traffic from outside Fluid Cache is interfering with performance.

Solution The cache network switch is not configured correctly. Configure the ports on the switch used by Fluid Cache so that they are used solely by Fluid Cache.

Possible Cause The cache network switch is not configured correctly.

Solution Make sure that the switch has the following settings:

The ports are in Layer 2 mode.

The ports are in an untagged state.

The switch firmware is up to date.

Multicast is enabled.

Verify that flow control is enabled and DCB is disabled. For Dell Networking switches, see Configuring the Cache Network Switch. For all other switches, consult the manufacturers documentation.

NOTE: Enabling flow control is a requirement for Fluid Cache.

For blade enclosure, disable FlexAddress.

You can check network functionality by checking rx_over_errors using the

ethtool -S command.

Possible Cause Fluid Cache is running on an evaluation license (typically 90 days) and that time period has been exceeded. You can still access data on cached volumes, but performance is degraded because the Fluid Cache cluster has been placed in maintenance mode and caching is no longer active.

Solution Check the status of the license file by selecting the Fluid Cache cluster in Enterprise Manager and referring to the status shown on the Events or Cache tabs. Contact your Dell representative to purchase a Fluid Cache license. After activating the new license, take the Fluid Cache cluster out of maintenance mode by selecting the cluster in Enterprise Manager, clicking Edit Settings in the Summary tab, and then clearing the Maintenance Mode option.

40

C Example Cabling Diagram

Figure 2. Example Cabling Diagram

1. Enterprise Manager console 2. Management interface switch

3. Servers (4) 4. Cache network switches (2)

5. SAN switch 6. Storage Center controllers (2)

7. Storage Center expansion enclosures (2)

The example above has the following features:

41

Fluid Cache is installed on four servers, which conforms to the minimum of three servers per each cache cluster and maximum of nine servers required.

Each of the servers has a network adapter connected to a port on each of the two cache network switches. The switches are uplinked to each other.

The servers access the Storage Center using SAN connectivity.

Enterprise Manager is used to configure and monitor Fluid Cache.

NOTE: For best practices on racking and cabling each hardware component, refer to the rack placement for the component at dell.com/support/manuals.

42

D Configuring a Dell Networking Switch The following procedure is for one of the supported Dell Networking switches listed in the table in Requirements For Fluid Cache. For all other switches, see the manufacturers documentation. To configure a switch containing physical ports 0, 1, 2, and 3 for a cache network with four nodes connected to the switchs 10-gigabit Ethernet ports:

1. Telnet into the switch and enter the login name and password to enter Exec mode.

While you are in Exec mode, the > prompt is displayed following the host name prompt, which is FTOS by default.

telnet 172.31.1.53 Trying 172.31.1.53... Connected to 172.31.1.53. Escape character is '^]'. Login: username Password: FTOS>

2. Enter Exec Privilege mode by running the command: enable 3. Enter Configuration mode by running the command: configure 4. Select either a single physical port or range of physical ports:

To configure a single port (in this example, port 0), run the command: interface TenGigabitEthernet 0/0

To configure a range of ports (in this example, ports 0-3), run the command: interface range TenGigabitEthernet 0/0 - 3

5. Define static IP addressing by running the command: no ip address 6. Enter Layer 2 mode by running the command: switchport 7. Enable the port by running the command: no shutdown 8. Enable flow control and disable DCB by running the command: flowcontrol rx on tx on

NOTE: Enabling flow control is a requirement for Fluid Cache.

9. Close the configuration page by running the command: exit 10. Repeat tasks 49 for the remaining physical ports in the cache network.

11. Exit Exec Privilege mode by running the command: exit 12. Save all the changes by running the command: write

NOTE: To create a VLAN (for example, to use available ports on an existing network switch to create the cache network), make sure that the cache network ports are in Layer 2 mode and in the untagged state.

43

44

E Checking the System Configuration If any of the system checks below show that the firmware or drivers are earlier than the required versions, update the components. See Requirements for Fluid Cache and Downloading Firmware, Drivers, and Software.

Checking the Cache Device Firmware and Driver

NVMe Cache Devices

Firmware Refer to the Dell PowerEdge Express Flash NVMe PCIe SSD Users Guide for instructions on checking your current firmware version.

Driver Refer to the Dell PowerEdge Express Flash NVMe PCIe SSD Users Guide for instructions on checking your current driver version.

Non-NVMe Cache Devices

Firmware Make sure the firmware is up to date by using this command: dmesg | grep mtip32xx | grep Firmware For each cache device, you must get results similar to the following, showing firmware version B1490508 or later:

mtip32xx 0000:46:00.0: Firmware Ver.: B1490508 NOTE: The firmware version must be B1490508 or later. Otherwise, even though they are visible in Enterprise Manager, the cache devices do not appear in the list of available devices when adding devices to a Fluid Cache cluster.

Driver Make sure the driver is up to date and running by using this command: dmesg | grep -i micron For each cache device, you must get results similar to the following, showing driver version 2.4.2 or later:

Micron RealSSD PCIe Block Driver Version 2.4.2

Checking the Network Adapter Firmware and Driver To make sure that the correct firmware and drivers are installed on each of the Mellanox network adapters used by Fluid Cache:

1. Get a list of interfaces on your server by running the command: ip addr 2. In the results, locate and write down the names of the network interfaces. These are often in the

format pXpY (for example, p6p2), but may be in the format ethX, emX, or other formats, depending on your operating system.

45

The following example illustrates potential Fluid Cache interfaces on Mellanox network adapters:

p6p2: mtu 1500 qdisc noop state DOWN qlen 1000 link/ ether a0:36:9f:14:12:a2 brd ff:ff:ff:ff:ff:ff

3. Check that you have the required OFED package installed by running the command: ofed_info 4. Make sure the firmware and driver versions on each of these interfaces by running the command:

ethtool -i <interface name> The firmware and driver must use supported versions as documented in the Release Notes.

The firmware and driver must use supported versions, as in this example:

driver: mlx4_en version: 2.6.8 (Aug 18 2013) firmware-version: 2.30.5118

5. Configure all interfaces for the management interface, the cache network, and potentially the iSCSI network.

6. Ping the management interface and cache network addresses from all nodes. To install Fluid Cache, you must be able to ping the management interface and cache network of every node in the cluster from each node in the cluster.

To download updated firmware or drivers, see Downloading Firmware, Drivers, and Software.

Checking iSCSI Status

If the backing store is iSCSI, make sure that you have logged in to your iSCSI targets by running these commands:

service iscsi status

service iscsid status In the output text, make sure that the state of the iSCSI hosts is displayed as running.

Checking Storage Center Connectivity

Make sure that the node can see the storage, specifically the Dell Compellent LUNs, by running this command: cat /proc/scsi/scsi You must see results similar to the following (abbreviated in this example):

Attached devices: Host: scsi0 Channel: 02 Id: 00 Lun: 00 Vendor: DELL Model: PERC H710 Rev: 3.13 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi7 Channel: 00 Id: 00 Lun: 02 Vendor: COMPELNT Model: Compellent Vol Rev: 0603 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi9 Channel: 00 Id: 00 Lun: 02 Vendor: COMPELNT Model: Compellent Vol Rev: 0603 Type: Direct-Access ANSI SCSI revision: 05

Checking Configured Servers

To check the servers configured for Fluid Cache, run the following command:

46

/opt/dell/fluidcache/bin/fldcsan --user=<username>:<password> --list hcn The results should be similar to the following, and should list all the servers you intend to include in the Fluid Cache cluster:

------------------------------------------------------- hcnId: 75fd0b3d-8626-4c1c-85da-3c2a2e518932 hccId: b8ddf6de-ba87-4ce9-9ab7-6c5b39dbf7af hostname: server1.company.com ------------------------------------------------------- hcnId: 70c152c5-8639-49e1-a870-a0fbb23b762d hccId: b8ddf6de-ba87-4ce9-9ab7-6c5b39dbf7af hostname: server2.company.com ------------------------------------------------------- hcnId: d22b054a-b3ff-4180-8b6e-1cc4169141a5 hccId: b8ddf6de-ba87-4ce9-9ab7-6c5b39dbf7af hostname: server3.company.com SUCCESS:

Checking Linux Dependencies

During installation, RPM checks for the necessary Linux dependencies and prompts you to install any that are not found.

To check which dependencies are currently installed on your system, run this command: rpm -qa

For a complete list of dependencies required for installation, refer to the Release Notes.

Avahi Functionality

Fluid Cache uses Avahi for autodiscovery between cluster nodes. To check that the Avahi daemon is running on a node, run this command: ps ef | grep avahi-daemon. To start the Avahi daemon, run

this command: /etc/init.d/avahi-daemon start. To ensure that the Avahi daemon starts when the

system is rebooted, run this command: chkconfig avahi-daemon on.

NOTE: If your avahi-daemon.conf file does not use the default settings, make sure that Disable Publishing is set to no, that Deny Interfaces does not list Fluid Cache interfaces, and that Allow Interfaces is either commented out or lists all Fluid Cache interfaces.

47

48

F Related Documentation For PowerEdge server documentation, go to dell.com/support/manuals and enter your Service Tag.

For cache device documentation, go to dell.com/storagecontrollermanuals and click Dell Power Edge Express Flash PCIeSSD.

The following table lists documents you may want to refer to while installing Fluid Cache.

Component Document Content

Server PowerEdge Owners Manual Describes how to install, remove, configure, and troubleshoot server components.

Rack Placement Describes how to rack the server.

Updating BIOS on Dell 12G PowerEdge Servers (iDRAC7 - 12G only)

Describes how to upgrade the BIOS on Dell 12G PowerEdge Servers using different Dell utilities.

Updating BIOS on Dell 13G PowerEdge Servers (iDRAC8 - 13G only) (New for 13G)

Describes how to upgrade the BIOS on Dell 13G PowerEdge Servers using different Dell utilities.

Lifecycle Controller Platform Update in Dell PowerEdge 12th Generation Servers

Describes how to upgrade the Lifecycle Controller using different Dell utilities.

Updating DRAC Firmware Describes how to upgrade iDRAC using different Dell utilities.

Cache Device PowerEdge Express Flash PCIeSSD Users Guide

Describes how to install, remove, configure, and troubleshoot PCIeSSDs.

Deploying the Dell PowerEdge Express Flash PCIeSSD

Describes the procedures for setting up, installing, and removing a PCIeSSD.

Network Switch Dell Networking Owners Manual/ Administrators Guide/ Rack Placement

Describes how to set up the switch.

Network Adapter ConnectX-3 VPI Adapter Card User Manual

Describes how to install, remove, and configure, the Ethernet adapter.

Dell Compellent Enterprise Manager

See Accessing Dell Compellent Documentation.

Release Notes Describes new features, known issues, and upgrade steps for Enterprise Manager.

Administrators Guide Describes how to monitor and run Dell Compellent Enterprise Manager.

49

Accessing Enterprise Manager and Storage Center Documentation Documentation for Dell Compellent products is not available at dell.com/support/manuals. To download Enterpris

Manualsnet FAQs

If you want to find out how the 2.0 Dell works, you can view and download the Dell Fluid Cache SAN 2.0 V2.0.0 Storage Deployment Guide For Linux Systems on the Manualsnet website.

Yes, we have the Deployment Guide For Linux Systems for Dell 2.0 as well as other Dell manuals. All you need to do is to use our search bar and find the user manual that you are looking for.

The Deployment Guide For Linux Systems should include all the details that are needed to use a Dell 2.0. Full manuals and user guide PDFs can be downloaded from Manualsnet.com.

The best way to navigate the Dell Fluid Cache SAN 2.0 V2.0.0 Storage Deployment Guide For Linux Systems is by checking the Table of Contents at the top of the page where available. This allows you to navigate a manual by jumping to the section you are looking for.

This Dell Fluid Cache SAN 2.0 V2.0.0 Storage Deployment Guide For Linux Systems consists of sections like Table of Contents, to name a few. For easier navigation, use the Table of Contents in the upper left corner.

You can download Dell Fluid Cache SAN 2.0 V2.0.0 Storage Deployment Guide For Linux Systems free of charge simply by clicking the “download” button in the upper right corner of any manuals page. This feature allows you to download any manual in a couple of seconds and is generally in PDF format. You can also save a manual for later by adding it to your saved documents in the user profile.

To be able to print Dell Fluid Cache SAN 2.0 V2.0.0 Storage Deployment Guide For Linux Systems, simply download the document to your computer. Once downloaded, open the PDF file and print the Dell Fluid Cache SAN 2.0 V2.0.0 Storage Deployment Guide For Linux Systems as you would any other document. This can usually be achieved by clicking on “File” and then “Print” from the menu bar.