Contents

Dell Fluid Cache SAN 2.1.0V2.1.0 Storage Deployment and Admin Guide For Linux Systems PDF

1 of 53
1 of 53

Summary of Content for Dell Fluid Cache SAN 2.1.0V2.1.0 Storage Deployment and Admin Guide For Linux Systems PDF

Dell Fluid Cache for SAN Version 2.1.0 Deployment and Admin Guide for Linux Systems

Notes, cautions, and warnings NOTE: A NOTE indicates important information that helps you make better use of your computer.

CAUTION: A CAUTION indicates either potential damage to hardware or loss of data and tells you how to avoid the problem.

WARNING: A WARNING indicates a potential for property damage, personal injury, or death.

2016 Dell Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. Dell and the Dell logo are trademarks of Dell Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.

2015 - 04

Rev. A00

Contents

1 Preface........................................................................................................................................ 6

Audience.......................................................................................................................................6

Related Documentation............................................................................................................6

Accessing Enterprise Manager and Storage Center Documentation........................8

Dell Online Services................................................................................................................... 8

Technical Support and Customer Service.............................................................................8

Contacting Dell........................................................................................................................... 8

2 Dell Fluid Cache for SAN Product Overview........................................................................9

Dell Fluid Cache for SAN Features.......................................................................................... 9

Fluid Cache for SAN Environment.......................................................................................... 9

Example Cabling Diagram................................................................................................. 11

Fluid Cache for SAN Deployment Requirements...............................................................12

Fluid Cache for SAN in Enterprise Manager GUI................................................................13

3 Preparing the Fluid Cache Components............................................................................ 14

Checking Security Settings..................................................................................................... 14

Checking Network Connections........................................................................................... 15

RDMA Functionality............................................................................................................ 15

Multipathing Support..........................................................................................................15

Checking Application Settings............................................................................................... 16

Preparing the Servers............................................................................................................... 16

Preparing the Cache Devices.................................................................................................16

NVMe Cache Devices.........................................................................................................16

Non-NVMe Cache Devices............................................................................................... 17

Preparing the Cache Network Adapters.............................................................................. 17

Bonding Network Adapter Ports......................................................................................18

Preparing the Cache Network Switch.................................................................................. 19

Configuring a Dell Networking Switch................................................................................ 20

4 Installing and Setting up Fluid Cache................................................................................. 21

Installing the Fluid Cache Software...................................................................................... 21

Setting up the Fluid Cache Servers....................................................................................... 21

5 Fluid Cache for SAN Cluster Creation and Management Operations...........................23

Creating a Fluid Cache Cluster..............................................................................................23

Managing a Fluid Cache Cluster Environment...................................................................25

Configuring Fluid Cache Volumes..................................................................................25

Deployment and Admin Guide for Linux Systems 3

Mapping Volumes in Fluid Cache....................................................................................25

Removing Volume Mappings...........................................................................................26

Removing Volumes............................................................................................................ 27

Deleting a Volume From a Fluid Cache Cluster...........................................................28

Adding Servers to a Fluid Cache Cluster....................................................................... 28

Removing a Server from a Fluid Cache Cluster........................................................... 29

Adding Cache Devices to a Fluid Cache Cluster......................................................... 29

Setting Storage Capacity for each Cache Server.........................................................30

Removing a Cache Device from a Fluid Cache Cluster.............................................30

Reactivate a Cache Device Attached to a Fluid Cache Cluster................................30

Reactivate a Volume on a Fluid Cache Cluster............................................................ 31

Adding a Storage Center to a Fluid Cache Cluster...................................................... 31

Remove a Storage Center from a Fluid Cache cluster............................................... 31

Reconnect a Fluid Cache Cluster to a Storage Center.............................................. 32

Change the License for a Fluid Cache cluster..............................................................32

Send Fluid Cache Cluster Information Using Dell SupportAssist............................. 32

Shutting Down a Fluid Cache Cluster............................................................................ 32

Enable Server Load Equalizing for Storage Center Volumes.................................... 33

Maintaining Fluid Cache Environments............................................................................... 33

Shutting Down and Restarting a Cluster....................................................................... 34

Uninstalling the Fluid Cache Software...........................................................................34

6 Fluid Cache Web Page Overview.........................................................................................35

Accessing Fluid Cache Web Page.........................................................................................35

Understanding Fluid Cache Web Page................................................................................ 35

Status.....................................................................................................................................35

CFM Info............................................................................................................................... 35

Peer CFMs............................................................................................................................ 36

Metadata Server Stats........................................................................................................ 36

Cache Server Stats..............................................................................................................36

Cache Server Per-Path Stats............................................................................................ 37

Block Device Stats.............................................................................................................. 38

A Troubleshooting Fluid Cache Installations....................................................................... 39

Basic Troubleshooting Steps................................................................................................. 40

Cannot Set up Fluid Cache on a Server.............................................................................. 40

Unable to remove a cache device from Fluid Cache Cluster using Enterprise

Manager...................................................................................................................................... 41

Incorrect Network Address or Netmask.............................................................................. 41

Server Does Not Appear in List of Servers.......................................................................... 42

Cache Device Does Not Appear in List of Cache Devices.............................................. 42

Cache Device Cannot Be Added to a Cluster.................................................................... 43

Cannot Select a Specific Cache Mode ............................................................................... 43

Deployment and Admin Guide for Linux Systems 4

Fluid Cache Node Is Unavailable in Enterprise Manager................................................. 44

Cached LUNs Are Unavailable...............................................................................................44

Cannot Create a Fluid Cache Cluster.................................................................................. 44

Fluid Cache License Is Expired.............................................................................................. 45

Fluid Cache License Is Invalid................................................................................................45

Cannot Configure the Cache Network............................................................................... 45

Cannot Assign or Remove a Storage Center......................................................................47

Cannot Determine Which Cache Device Failed................................................................ 47

Events for Fluid Cache Are Not Shown in Enterprise Manager...................................... 47

Cluster or Application Has Performance Issues................................................................ 48

Unable to Recreate a Fluid Cache Cluster After a Hardware Failure............................ 49

Cache Node is Not Listed.......................................................................................................49

Unable to Select a Specific Caching Mode........................................................................ 49

Fluid Cache License File is Invalid.........................................................................................50

Option to Create Cluster Not Available...............................................................................50

Unable to Add a Volume to a Fluid Cache Cluster........................................................... 50

Event Messages Are Not Being Delivered........................................................................... 50

Storage Center is Not Available............................................................................................. 51

Fluid Cache Server is Not Available.......................................................................................51

Information Displays Differently Between Storage Centers and Fluid Cache

Clusters....................................................................................................................................... 51

Verify That All Parts of the Fluid Cache Cluster are Communicating with Each

Other............................................................................................................................................51

Verify the Data Path is Working............................................................................................. 52

The Cluster in the Fluid Cache Clusters Display is Marked Red..................................... 52

Problems Configuring Server Clusters Defined on a Storage Center with Dell

Fluid Cache for SAN.................................................................................................................52

Deployment and Admin Guide for Linux Systems 5

Preface Dell Fluid Cache for SAN is a server-side caching accelerator software. Fluid Cache makes high-speed PCI Express (PCIe) SSDs a shared, distributed cache resource. Fluid Cache is deployed on clusters of Dell PowerEdge systems connected using RoCE-enabled Ethernet adapters and operates within a SAN environment employing a Dell Compellent backing store.

This guide provides information on how to install, configure and manage a Fluid Cache for SAN 2.1.0 for Linux based environments.

Topics:

Audience

Related Documentation

Dell Online Services

Technical Support and Customer Service

Contacting Dell

Audience This guide is intended for Dell Enterprise Deployment team (EDT), Customers, Channel Partners responsible for the deployment, configuration and administration of Fluid Cache for SAN 2.1.0 environments.

The assumption in this guide is that you have sufficient understanding on PowerEdge servers, networking configuration, management of Dell Storage or Compellent, and good understanding of tiered storage with Dell Storage.

Related Documentation For PowerEdge server documentation, go to Dell.com/support/manuals and enter your Service Tag.

For cache device documentation, go to Dell.com/storagecontrollermanuals and click Dell Power Edge Express Flash PCIeSSD.

For Fluid Cache documentation, go to ell.com/CacheSolutions.

The following table lists documents you may want to refer to while installing and managing Fluid Cache.

1

Preface 6

Component Document Content

Fluid Cache Release Notes Describes new features, known issues, and upgrade steps for Enterprise Manager.

Compatibility Matrix Lists the compatibility matrix of different components included in the Dell Fluid Cache for SAN infrastructure.

Server PowerEdge Owners Manual Describes how to install, remove, configure, and troubleshoot server components.

Rack Placement Describes how to rack the server.

Updating BIOS on Dell 12G PowerEdge Servers (iDRAC7 - 12G only)

Describes how to upgrade the BIOS on Dell 12G PowerEdge Servers using different Dell utilities.

Updating BIOS on Dell 13G PowerEdge Servers (iDRAC8 - 13G only) (New for 13G)

Describes how to upgrade the BIOS on Dell 13G PowerEdge Servers using different Dell utilities.

Lifecycle Controller Platform Update in Dell PowerEdge 12th Generation Servers

Describes how to upgrade the Lifecycle Controller using different Dell utilities.

Updating DRAC Firmware Describes how to upgrade iDRAC using different Dell utilities.

Cache Device PowerEdge Express Flash PCIe SSD Users Guide

Describes how to install, remove, configure, and troubleshoot PCIeSSDs.

Deploying the Dell PowerEdge Express Flash PCIe SSD

Describes the procedures for setting up, installing, and removing a PCIeSSD.

Network Switch Dell Networking Owners Manual/ Administrators Guide/ Rack Placement

Describes how to set up the switch.

Network Adapter ConnectX-3 VPI Adapter Card User Manual

Describes how to install, remove, and configure, the Ethernet adapter.

Dell Compellent Enterprise Manager

See Accessing Dell Compellent Documentation .

Release Notes Describes new features, known issues, and upgrade steps for Enterprise Manager.

Administrators Guide Describes how to monitor and run Dell Compellent Enterprise Manager.

Preface 7

Accessing Enterprise Manager and Storage Center Documentation Documentation for Dell Compellent products is not available at dell.com/support/manuals. To download Enterprise Manager and Storage Center documentation:

1 Go to portal.compellent.com.

2 Enter your user name and password and click Login.

If you do not have a registration, send an email to customer.portal@compellent.com.

3 In the portal page, click Knowledge Center.

4 Under Product in the left pane, select either Enterprise Manager or Storage Center, and download the documents.

Dell Online Services You can learn about Dell products and services using this procedure:

1 Visit www.dell.com or the URL specified in any Dell product information.

2 Use the locale menu or click the link that specifies your country or region.

Technical Support and Customer Service Dells support service is available to answer your questions about Fluid Cache for SAN 2.1.0.

Contacting Dell NOTE: If you do not have an active Internet connection, you can find contact information on your purchase invoice, packing slip, bill, or Dell product catalog.

Dell provides several online and telephone-based support and service options. Availability varies by country and product, and some services may not be available in your area. To contact Dell for sales, technical support, or customer service issues:

1 Go to dell.com/support.

2 Select your support category.

3 Verify your country or region in the Choose a Country/Region drop-down list at the bottom of the page.

4 Select the appropriate service or support link based on your need.

Preface 8

Dell Fluid Cache for SAN Product Overview

Dell Fluid Cache for SAN is a server-side caching accelerator software. Fluid Cache makes high-speed PCI Express (PCIe) SSDs a shared, distributed cache resource. Fluid Cache is deployed on clusters of Dell PowerEdge systems connected using RoCE-enabled Ethernet adapters and operates within a SAN environment employing a Dell Compellent backing store.

Topics:

Dell Fluid Cache for SAN Features

Fluid Cache for SAN Environment

Fluid Cache for SAN Deployment Requirements

Fluid Cache for SAN in Enterprise Manager GUI

Dell Fluid Cache for SAN Features Fluid Cache supports the following key features:

Write-back caching.

Accelerates reads, writes, and read-after writes.

Up to nine nodes in the cache cluster.

Dynamic cache expansion.

Centralized configuration, management, and reporting.

Fluid Cache for SAN Environment Fluid Cache operates on the following networks and interfaces:

Cache Network A high speed, low latency private network that Fluid Cache uses to connect the servers in the Fluid Cache cluster.

Management Interface

The connection to Dell Compellent Enterprise Manager, which manages Fluid Cache and the SAN.

2

Dell Fluid Cache for SAN Product Overview 9

Storage Area Network

The network that Dell Compellent Storage Center uses to handle data connectivity within the SAN.

Figure 1. Fluid Cache Connectivity

1 Fluid Cache nodes communicate with each other over a private network using RDMA.

2 Enterprise Manager creates, manages, and monitors the Fluid Cache clusters.

3 Fluid Cache nodes communicate with the Management IP (VIP) of the Storage Controllers.

4 Enterprise Manager manages the Dell Compellent array.

The interfaces illustrated in the figure are associated with specific ports (see Checking Network Connections). For a schematic of an example Fluid Cache installation, see Example Cabling Diagram.

Dell Fluid Cache for SAN Product Overview 10

Example Cabling Diagram Figure 2. Example Cabling Diagram

1 Enterprise Manager console 2 Management interface switch

3 Servers (4) 4 Cache network switches (2)

5 SAN switch 6 Storage Center controllers (2)

7 Storage Center expansion enclosures (2)

The example above has the following features:

Dell Fluid Cache for SAN Product Overview 11

Fluid Cache is installed on four servers, which conforms to the minimum of three servers per each cache cluster and maximum of nine servers required.

Each of the servers has a network adapter connected to a port on each of the two cache network switches. The switches are uplinked to each other.

The servers access the Storage Center using SAN connectivity.

Enterprise Manager is used to configure and monitor Fluid Cache.

NOTE: For best practices on racking and cabling each hardware component, refer to the rack placement for the component at dell.com/support/manuals.

Fluid Cache for SAN Deployment Requirements

The following are prerequisites for deploying Fluid Cache for SAN. Refer to the Release Notes for the most recent requirements.

The supported versions of firmware, drivers, and software are required to run Fluid Cache. These are listed in the Release Notes and are available at dell.com/support.

For the current list of validated hardware and software components, see the Dell Fluid Cache for SAN Compatibility Matrix available at http://www.dell.com/CacheSolutions.

Table 1. System Requirements

Servers The cluster on which you deploy Fluid Cache must contain a minimum of three server per each cache cluster and a maximum of nine servers.

Cache Devices There must be a cache device installed on at least two servers in the cluster.

Network Adapters A network adapter that supports Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE) must be installed on each server in the cluster.

Network Switch A compatible switch must be available for the cache network. A redundant switch configuration is recommended.

SAN Management Software

Enterprise Manager must be started and configured to manage the Storage Center in use with Fluid Cache.

SAN Connectivity All servers in the cluster must be connected to the SAN and be displayed on the Dell Compellent array.

Dell Fluid Cache for SAN Product Overview 12

Fluid Cache for SAN in Enterprise Manager GUI Use the Enterprise Manager Graphical User Interface (GUI) to centrally manage your Dell Fluid Cache for SAN clusters and monitor the status and performance of the Dell Fluid Cache for SAN cluster.

Dell Fluid Cache for SAN Product Overview 13

Preparing the Fluid Cache Components

Before installing Fluid Cache, you must prepare the components of the Fluid Cache network: the servers, cache devices, network cards, and switches.

The instructions that follow assume that you have an existing SAN configured and managed. All nodes in the cache cluster must be connected to the SAN and visible on the Dell Compellent array. Instructions for racking and cabling a SAN solution are beyond the scope of this document. For more information about installing, upgrading, and configuring specific Dell components, see Related Documentation.

Topics:

Checking Security Settings

Checking Network Connections

Checking Application Settings

Preparing the Servers

Preparing the Cache Devices

Preparing the Cache Network Adapters

Preparing the Cache Network Switch

Configuring a Dell Networking Switch

Checking Security Settings Before you start the installation, make sure that:

If you are using servers that supports SSL v2, to prevent the servers from DROWN attack, upgrade the OpenSSL version to OpenSSL 1.0.1e.

Firewall settings do not block traffic or devices on the cache network, or on networks used by Fluid Cache (refer to the table of Port Specifications below).

SELinux, AppArmor, or other security frameworks do not limit applications or devices on the cache network.

iptables exceptions have been entered where appropriate. (Note that some default RHEL installations may create an iptables entry that blocks Fluid Cache. See the troubleshooting topic Cannot Configure Fluid Cache.

If your avahi-daemon.conf file does not use the default settings, make sure that Disable Publishing is set to no, Deny Interfaces does not list Fluid Cache interfaces, and Allow Interfaces is either commented out or lists all Fluid Cache interfaces.

NOTE: If any of these settings are incorrect, the Fluid Cache cluster cannot be configured.

3

Preparing the Fluid Cache Components 14

Checking Network Connections For Fluid Cache to function correctly, each Fluid Cache server must be able to communicate with other network components. Make sure that the following ports are available:

Table 2. Ports Specifications

Interface Port number Port type

Fluid Cache cluster Configuration Manager 7449 TCP

Node Auto-Discovery functionality 5353 UDP

Web-based server status tool 6773 and 6774 TCP (https)

(optional) Web-based server status tool 8080 and 8082 TCP (http)

Fluid Cache to Enterprise Manager 3033 TCP

Fluid Cache to Storage Center 3033 TCP

NOTE: Fluid Cache cannot be configured and does not function properly if any of these ports are unavailable. To diagnose network connection problemsand reassign port numbers, see the troubleshooting topic Cannot Configure Fluid Cache.

RDMA Functionality To make sure that RDMA is working correctly between the cluster nodes, use the utility ib_send_bw:

1 On one node, run the command: ib_send_bw 2 On a second node, run the command: ib_send_bw <IP_address_of_first_node>

If the test is successful, the bandwidth for the node displays on each node.

Multipathing Support Fluid Cache supports multipathing and uses existing multipath configurations. You must make two modifications to the Devices section of the /etc/multipath.conf file on each node in the cluster:

1 For the vendor COMPELNT, specify no_path_retry 6 2 Add blacklist entries to exclude devices used by Fluid Cache, as shown:

blacklist { devnode "^fldc[0-9]*" devnode "^nvme[a-z]*" devnode "^rssd[a-z]*" }

Preparing the Fluid Cache Components 15

Checking Application Settings All applications that use volumes mapped to Fluid Cache must be configured to start after Fluid Cache and exit before Fluid Cache.

Preparing the Servers Make sure that each server in the Fluid Cache cluster has the latest supported BIOS version, Lifecycle Controller firmware, and iDRAC firmware.

For updating Dell Lifecycle Controller and BIOS firmware on 13th generation of PowerEdge servers, see the Dell Lifecycle Controller Graphical User Interface Version 2.05.05.05 For 13th Generation Dell PowerEdge Servers User's Guide, available at http://topics-cdn.dell.com/pdf/idrac8-with-lc- v2.05.05.05_User's%20Guide2_en-us.pdf

For updating Dell iDRAC firmware on 13th generation of PowerEdge servers, see the Integrated Dell Remote Access Controller 8 (iDRAC8) Version 2.05.05.05 User's Guide, available at http://topics- cdn.dell.com/pdf/idrac8-with-lc-v2.05.05.05_User's%20Guide_en-us.pdf

For other latest information related to Dell iDRAC and Dell Lifecycle Controller documentation, go to http://www.dell.com/support/home/us/en/19/product-support/product/idrac8-with-lc- v2.05.05.05/research

Modify your BIOS settings as follows:

1 Enter the system BIOS and set System Profile Settings to Performance.

2 Click Back, and then click OK to return to the main BIOS page.

3 Restart the server to activate the BIOS changes.

4 Repeat this process for each server in the Fluid Cache cluster.

Preparing the Cache Devices Make sure that all cache devices used in the Fluid Cache cluster have the minimum firmware and driver

versions specified in Requirements for Fluid Cache. To download updated firmware or drivers.

Fluid Cache supports clusters containing cache devices of different sizes, but for the most predictable performance, it is recommended that the cache devices be of similar capacity.

NVMe Cache Devices Firmware Refer to the Dell PowerEdge Express Flash NVMe PCIe SSD Users Guide for instructions about

checking your current firmware version. Use the Dell Update Package to install the latest firmware.

Driver Refer to the Dell PowerEdge Express Flash NVMe PCIe SSD Users Guide for instructions about checking your current driver version and installing the updated driver.

Preparing the Fluid Cache Components 16

Non-NVMe Cache Devices Firmware Make sure the firmware is up to date by using this command: dmesg | grep mtip32xx |

grep Firmware For each cache device, you must get results similar to the following, showing firmware version B1490508 or later:

mtip32xx 0000:46:00.0: Firmware Ver.: B1490508

NOTE: The firmware version must be B1490508 or later. Otherwise, even though they are visible in Enterprise Manager, the cache devices do not appear in the list of available devices when adding devices to a Fluid Cache cluster.

Driver Make sure the driver is up to date and running by using this command: dmesg | grep -i micron For each cache device, you must get results similar to the following, showing driver version 2.4.2 or later:

Micron RealSSD PCIe Block Driver Version 2.4.2

CAUTION: If the cache devices contain data, back up this data before adding the cache devices to the Fluid Cache cluster. All data on the cache devices are lost when they are added to the cluster.

NOTE: If your server already has cache devices installed, make sure that the cache devices you intend to use in the Fluid Cache cluster are not used by any other application.

Preparing the Cache Network Adapters 1 Install the network adapters in PCIe slots on the servers (x16 slots, if available).

2 Check the network adapter driver and firmware versions, and the OFED package version, and then update, if necessary.

NOTE: It is recommended to use an MTU of 1500 for RDMA or Cache Network adapter connectivity.

NOTE: Do not use a network adapter driver downloaded from the manufacturers site. Fluid Cache uses a custom driver available only through Dell.

3 Create and configure an IP address on all RoCE network interfaces that are connected:

a Open the interface configuration file by running the commands:

cd /etc/sysconfig/network-scripts vi ifcfg-

b Edit the configuration parameters as given in the example here:

DEVICE=" " BOOTPROTO="static" ONBOOT="yes" IPADDR="<ip address of the network adapter>" NETMASK="<netmask of the cache network>" NAME=" "

c Make a note of the address and netmask for later reference when configuring the cache network.

d Save your changes to the configuration file.

e Open the interface by running the command: ifup 4 If port bonding is used, refer to Bonding Network Adapter Ports.

Preparing the Fluid Cache Components 17

5 If a blade enclosure is used, disable FlexAddress in the blade enclosure.

6 Repeat this process for each network adapter in the cache network.

NOTE: Make sure that all RoCE network adapters used by the Fluid Cache network are dedicated to the cache network and are not configured for any other network traffic.

Bonding Network Adapter Ports Fluid Cache supports port bonding in active/passive mode (also called active/backup or master/slave).

Before implementing bonding on your network adapters, make sure that your cache network switches and cabling are correctly configured for bonding. For an example of a system cabled for port bonding, see Example Cabling Diagram.

NOTE: It is recommended that port bonding be used in configurations with redundant, uplinked cache network switches.

The following example describes configuring port bonding on an RHEL system using Dell Networking switches. For non-Dell Networking switches refer to the documentation specific to that networking switch.

1 Configure a virtual interface to be used as a bonded port (in this example, bond0) by running the following command:

vi /etc/modprobe.d/bond0.conf 2 In the configuration file, edit the parameters as follows:

alias netdev-bond0 bonding 3 Open the configuration file for the bonded port by running the following commands:

cd /etc/sysconfig/network-scripts

vi ifcfg-bond0 4 Edit the parameters in the configuration file as given in the example here:

DEVICE="bond0" BOOTPROTO="static" ONBOOT="yes" IPADDR="<ip address of the network adapter>" NETMASK="<netmask of the cache network>" NAME="bond0" BONDING_OPTS="mode=1 fail_over_mac=1 miimon=100 downdelay=300 updelay=300"

NOTE: In the BONDING_OPTS line, the values for mode and fail_over_mac must be set exactly as

shown in the example.

5 Configure an interface for use on the bonded port by running the following command:

vi /etc/sysconfig/network-scripts/ifcfg-<interface name>

NOTE: For the names of the interfaces on your server used by Fluid Cache, run the command: ip addr. The names of the network interfaces used by Fluid Cache are often in the format pXpY (for

example, p6p2), but may be in the format ethX, emX, or other formats, depending on your operating

system.

Preparing the Fluid Cache Components 18

6 In the configuration file, edit the parameters as follows:

DEVICE=<interface name> BOOTPROTO="none" ONBOOT="yes" NM_CONTROLLED="no" SLAVE="yes" MASTER="bond0"

7 For the other interface in the bonded port, repeat the tasks in 56.

8 Start network connection to the bonded port by running the following command: ifup bond0 9 Check the status of your bonded port and its interfaces by running the following command: ifconfig

The output must contain entries similar to the following. Note the line that states that the port session is started as a master or slave, and note that the slave interfaces no longer have individual IP addresses:

bond0 Link encap:Ethernet HWaddr 00:02:C9:E5:C5:C0 inet addr:172.20.3.116 Bcast:172.20.7.255 Mask:255.255.248.0 inet6 addr: fe80::202:c9ff:fee5:c5c0/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 . . . p5p1 Link encap:Ethernet HWaddr 00:02:C9:E5:C5:C0 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 . . . p5p2 Link encap:Ethernet HWaddr 00:02:C9:E5:C5:C0 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 . . .

10 Open the avahi-daemon.config file, and in the Server section, add this entry: Allow Interfaces = bond0.

11 Make sure that all servers in the cache network can ping every other server, and also the management interface.

Preparing the Cache Network Switch For a schematic of an example Fluid Cache installation, see Example Cabling Diagram.

For configuration tasks for a supported Dell Networking switch, see Configuring a Dell Networking Switch. For all other switches, see the manufacturers documentation.

For a list of supported switches, see Requirements for Fluid Cache.

1 Connect the ports on the network adapter to properly configured ports on a network switch.

NOTE: A redundant switch configuration is recommended.

If you have two switches for redundancy, on all of the cache network adapters, you must connect port 1 to one switch and port 2 to the other switch.

2 Make sure that all ports used in the cache network have the following settings:

The ports are in layer 2 mode.

The ports are in an untagged state.

The switch firmware is up to date.

Preparing the Fluid Cache Components 19

Flow control (transmit and receive) is enabled and Data Center Bridging (DCB) is disabled.

NOTE: Enabling flow control is a requirement for Fluid Cache.

3 Save the running configuration.

4 To implement the changes, restart the switch.

Configuring a Dell Networking Switch The following procedure is for one of the supported Dell Networking switches listed in the table in Requirements For Fluid Cache. For all other switches, see the manufacturers documentation. To configure a switch containing physical ports 0, 1, 2, and 3 for a cache network with four nodes connected to the switchs 10-gigabit Ethernet ports:

1 Telnet into the switch and enter the login name and password to enter Exec mode.

While you are in Exec mode, the > prompt is displayed following the host name prompt, which is FTOS by default.

telnet 172.31.1.53 Trying 172.31.1.53... Connected to 172.31.1.53. Escape character is '^]'. Login: username Password: FTOS>

2 Enter Exec Privilege mode by running the command: enable 3 Enter Configuration mode by running the command: configure 4 Select either a single physical port or range of physical ports:

To configure a single port (in this example, port 0), run the command: interface TenGigabitEthernet 0/0

To configure a range of ports (in this example, ports 0-3), run the command: interface range TenGigabitEthernet 0/0 - 3

5 Define static IP addressing by running the command: no ip address 6 Enter Layer 2 mode by running the command: switchport 7 Enable the port by running the command: no shutdown 8 Enable flow control and disable DCB by running the command: flowcontrol rx on tx on

NOTE: Enabling flow control is a requirement for Fluid Cache.

9 Close the configuration page by running the command: exit 10 Repeat tasks 49 for the remaining physical ports in the cache network.

11 Exit Exec Privilege mode by running the command: exit 12 Save all the changes by running the command: write

NOTE: To create a VLAN (for example, to use available ports on an existing network switch to create the cache network), make sure that the cache network ports are in Layer 2 mode and in the untagged state.

Preparing the Fluid Cache Components 20

Installing and Setting up Fluid Cache Before completing the tasks in this section, install the required Linux dependencies.

Topics:

Installing the Fluid Cache Software

Setting up the Fluid Cache Servers

Installing the Fluid Cache Software 1 Copy to the server the Fluid Cache tar.gz package that you downloaded earlier.

2 Expand the tar.gz package.

A new Fluid Cache directory is created, which contains an RPM file.

3 Change to the new Fluid Cache directory and run the following command:

rpm i <RPM file name>

There is a delay while the Fluid Cache software is installed on the server. During this process, RPM checks for the necessary Linux dependencies and asks you to install any that are not found. For a list of required dependencies, refer to the Release Notes available at the support site.

When processing is complete, the command prompt is displayed again.

NOTE: You can also install the software with a Yum repository using the following command: yum install <RPM file name>

4 Repeat tasks 13 for each server in the Fluid Cache cluster.

Setting up the Fluid Cache Servers Before setting up the servers in your Fluid Cache cluster:

Make sure that you have IP addresses configured for the network adapters on each server.

Make sure that you can ping the Mellanox interfaces on every server in the Fluid Cache cluster.

Record the following information for your servers (contact a network administrator, if necessary):

Management interface (used by Enterprise Manager to manage the SAN)

Network address: ________________

Netmask: ______________________

Cache network

Network address: ________________

4

Installing and Setting up Fluid Cache 21

Netmask: ______________________

An example of the required information is a device with an IP address of 172.18.1.2, whose network

address is 172.18.1.0, and netmask is 255.255.255.0.

1 Change to the following directory:/opt/dell/fluidcache/bin/ 2 Start the Host Cache Node (HCN) Setup tool by running the following command:

./hcn_setup.py

HCN Setup sets up a server for use as a Fluid Cache cluster node, and starts an agent on the server that allows it to be discovered by Enterprise Manager.

3 Press to begin the setup process.

A dialog box appears for a network address (for example, 172.18.1.0) and netmask (for example,

255.255.255.0) for the interface used by Enterprise Manager to manage the SANthe management

interface.

4 Enter the network address and press .

5 Enter the netmask and press .

6 When prompted to confirm your entries, press .

7 If you are using more than one management network, enter the address and subnet mask for the auxiliary network. Otherwise, press .

8 On the Cache Network page, enter the network address and press .

9 Enter the netmask and press .

10 When prompted to confirm your entries, press .

A message is displayed indicating that configuration was successful and prompting you to start Fluid Cache.

11 Press to start the Fluid Cache software.

A message is displayed indicating that the server is configured for Fluid Cache.

12 Repeat this process for the other servers in the Fluid Cache cluster.

You must set up Fluid Cache on at least three servers before creating a Fluid Cache cluster.

NOTE: If configuration is unsuccessful because of an incorrect value for a network address or netmask, restart HCN Setup and enter the correct values.

After setting up the Fluid Cache servers, create and configure Fluid Cache clusters in Enterprise Manager.

NOTE: HCN Setup also supports a CLI unattended installation. The commands used are:

hcn_setup.py -i <management interface IP address>:<subnet mask> -I <cache network IP address>:<subnet mask> -u [for unattended installations] -r [to run agent]

Installing and Setting up Fluid Cache 22

Fluid Cache for SAN Cluster Creation and Management Operations

After you configure and validate the Fluid Cache for SAN components, use Enterprise Manager to create and manage the Fluid Cache Environment as described in the following sections.

Creating a Fluid Cache Cluster. See Creating a Fluid Cache Cluster

Managing a Fluid Cache Cluster Environment. See Managing a Fluid Cache Cluster Environment

Maintaining a Fluid Cache Cluster Environment. See Maintaining Fluid Cache Environments

Topics:

Creating a Fluid Cache Cluster

Managing a Fluid Cache Cluster Environment

Maintaining Fluid Cache Environments

Creating a Fluid Cache Cluster At least one cache server must be installed, cabled, and able to communicate with a Storage Center

running version 6.5.1 or later.

To make the cluster fully redundant, use at least three servers.

To use write-back caching, at least two cache devices must be present in the cluster and installed in separate servers.

To use write-through caching, at least one cache device must be present in the cluster.

Your Enterprise Manager user account must have the Administrator or Volume Manager privilege. In addition, you must have admin (root) access to at least one of the cache servers.

The Fluid Cache license file must reside on the system running the Dell Storage Client and is used to create the cluster, or a share available to it.

At least one Storage Center must be added to Enterprise Manager.

1 Log in to Enterprise Manager as a user with Administrator or Volume Manager privileges.

2 In Enterprise Managers Storage view, expand Storage Centers if necessary, and then select a Storage Center.

3 In the Storage pane, select Dell Storage.

4 In the Summary tab, click Configure Fluid Cache Cluster to launch the configuration wizard. (Do not click Add FluidFS Cluster.)

The Discover Fluid Cache Servers page of the configuration wizard is displayed.

5 Type or select appropriate data in the Discover Fluid Cache Servers window.

5

Fluid Cache for SAN Cluster Creation and Management Operations 23

a In the Host or IP Address box, type the host name or IP address associated with the management network of any available Fluid Cache server.

b The Port box is autopopulated. Change only if necessary.

c In the User Name box, type the username , which is fldc.

You can also use the root user name and password in these boxes, if available.

d In the User Password box, type the password . The default value is calvin.

NOTE: If you change these default properties on any node in a Fluid Cache cluster, you must change them on all nodes. The login and password must be the same on all nodes in the Fluid Cache cluster.

e Click Next.

The Select Servers window is displayed.

6 By default, all available servers are selected. Clear the check box next to unwanted servers or select the Unselect All option, and then select three or more servers to be included. (Click Select All to use all available servers again.)

7 Click Next.

The Cluster Settings page of the configuration wizard is displayed.

8 Type or select appropriate data in the Cluster Settings window.

a In the Name box, enter a name for the cluster.

b Click Browse next to the License File box.

The Select Fluid Cache License File dialog box is displayed.

c Browse to the location of the license file, select the file, and then click Save.

d Verify that the license file and path displayed are correct and click Next.

The system processes for a few minutes while the cluster is created and the system automatically identifies compatible servers and devices on the network. After this process is completed, the Select Devices window is displayed.

9 By default, all available Fluid Cache devices are selected. Clear the check box next to unwanted devices or select the Unselect All option, and then select the required devices. (Select the Select All option to use all available devices again.)

CAUTION: You will lose any existing data on the cache devices when they are added to the Fluid Cache cluster. Back up this data before proceeding.

10 Click Next.

The Select Storage Centers page of the configuration wizard is displayed.

11 In the Select Storage Centers page, select one or more Storage Centers to include in the Fluid Cache cluster, and then click Finish.

After a delay while the system processes, the Storage pane contains a new top-level folder named Fluid Cache Clusters. Inside this folder is the Fluid Cache cluster that was just created. Fluid Cache clusters are denoted by a blue circle with the letters FC.

Select the Fluid Cache cluster. In the Summary tab, different areas of the window show the cluster's status, servers, devices, and other information.

After creating a Fluid Cache cluster, map volumes to the cluster. See Mapping Volumes.

Fluid Cache for SAN Cluster Creation and Management Operations 24

Managing a Fluid Cache Cluster Environment

Configuring Fluid Cache Volumes A Fluid Cache volume extends a normal Storage Center volume to be contained across the cache devices in a Fluid Cache cluster as well as permanently stored in the Storage Center volume.

Limitations for Fluid Cache Volumes There are a number of considerations before utilizing a volume in a Fluid Cache cluster:

Fluid Cache volumes cannot be expanded

Fluid Cache volume Read and Write Cache options are not available

Fluid Cache volumes cannot be the source or destination of Live Volumes

Fluid Cache volumes cannot be the destination of a replication

Fluid Cache volumes cannot be part of a synchronous replication or simulation

Mapping Volumes in Fluid Cache In Enterprise Manager, volume mappings created for a server in a Fluid Cache cluster behave similar to any other volume mappings. Volumes can be mapped either to an individual server, or to a server cluster (a subcluster) within the Fluid Cache cluster.

NOTE: If you have a shared data application, such as a clustered file system or clustered application, map volumes to a subcluster.

NOTE: Any applications that were using a volume before it was cached must be modified to use the new cached volume. When a cached volume is mapped to one or more servers, Fluid Cache creates its own Linux device file, /dev/fldcX. (The first device mapped is /dev/fldc0, the second is /dev/fldc1,

and so on.) Any application that uses the cached volume must be configured to use the new Fluid Cache device rather than the /dev/sdX or /dev/mpathX device. If the cached volume is shared by multiple

servers, the /dev/fldcX device is available on each server.

Mapping Volume to Servers Before mapping a volume to a server node, create a Fluid Cache cluster (see Creating a Fluid Cache Cluster).

To map a volume to a server:

1 In the Storage view, expand Storage Centers if necessary, and then select the Storage Center that contains the appropriate volume.

2 In the Storage tab, expand Volumes if necessary, and then locate the volume you want to map.

Fluid Cache for SAN Cluster Creation and Management Operations 25

3 Right-click the volume and select Map Volume to Server.

4 In the Map Volume to Server window, select the server.

5 Click Next.

6 Select Enable Fluid Cache.

7 From the Host Cache Policy drop-down menu, select a cache mode:

Write-back (default): In addition to caching reads, write-back mode allows the caching of written data without waiting for the Compellent Array to acknowledge the write operation. Write-back caching requires a cache device on two or more servers in the cluster.

Write-through: Write-through mode forces writes to both the cache and the Compellent Array simultaneously. Warm reads and read-after writes are accelerated but write operations are not. Write-through caching requires only one cache device on one server in the cluster.

NOTE: The cache mode selected for a volume mapping cannot be changed. To select a different cache mode, you must remove the mapping and create a new mapping. When you create the new mapping, you can then select a different cache mode.

8 (Optional) Select the Keep cached data on the node that accessed the data option. Selecting this option gives the best performance for warm reads from the cache for datasets that fit on the local nodes cache devices, because all data is local to the client. If this option is not selected, cached data is evenly distributed among all cache devices.

9 Click Finish.

At this stage, there is a delay while the system makes a number of configuration changes.

Mapping Volumes to a Subcluster Before mapping volumes to a subcluster, create a Fluid Cache cluster (see Creating a Fluid Cache Cluster) and a server cluster (subcluster) within it. To map a volume to a subcluster, follow the procedure for mapping a volume to a server (see previous section), but instead of selecting a server in the Map Volume to Server window (step 4), select a subcluster within a Fluid Cache cluster.

Although not specific to Fluid Cache, be aware of these aspects of volume mappings created at the subcluster level in Enterprise Manager:

When servers are added to or removed from the subcluster, they automatically inherit or disinherit the subclusters volume mappings.

You can promote a volume mapping from a server to the subcluster, and demote it from the cluster to the subcluster.

If a server has existing mappings, the server keeps those mappings when it becomes part of a subcluster.

Removing Volume Mappings To see a list of Fluid Cache mappings, select the appropriate Storage Center in the Storage view, and in the in the Storage tab, select the Fluid Cache cluster. The Fluid Cache mappings for the cluster are listed in the Volumes area at the bottom of the page.

Fluid Cache for SAN Cluster Creation and Management Operations 26

Removing Volume Mappings from a Server 1 Make sure the volume is no longer in use.

2 In Enterprise Managers Storage view, expand Storage Centers if necessary and select the appropriate Storage Center. (Do not select Fluid Cache Clusters or its contents.)

3 In the Storage tab, expand Servers if necessary and locate the server whose Fluid Cache mappings you want to remove.

4 Right-click the server and select Remove Mappings.

The Remove Mappings window is displayed.

5 In the Remove Mappings window, select the volume and click OK.

There may be some delay while the mapping is removed and dirty data in the cache is flushed to main storage. When the mapping removal is complete, the volumes icon in the Storage tab turns from blue to gray color. You may need to manually refresh the display.

NOTE: The Fluid Cache mappings for the server are listed in the Volumes area at the bottom of the page.

Removing Volume Mappings from a Subcluster 1 In Enterprise Managers Storage view, expand Storage Centers if necessary and select the appropriate

Storage Center. (Do not select Fluid Cache Clusters or its contents.)

2 In the Storage tab, expand Servers, and then the Fluid Cache clusters if necessary and select the subcluster whose mappings you want to remove.

3 In the right pane, below the list of servers, select the volume and click Remove Mappings.

There may be some delay while the mapping is removed and dirty data in the cache is flushed to main storage. When the mapping removal is complete, the volumes icon in the Storage tab turns from blue to gray color. You may need to manually refresh the display.

NOTE: The Fluid Cache mappings for the subcluster are listed in the Volumes area at the bottom of the page.

NOTE: All Fluid Cache volume mappings must be removed before a server can be removed from a Fluid Cache cluster.

Removing Volumes To avoid the potential for performance issues and data loss, it is a recommended best practice to unmount, remove, and delete all volume information after the volume is no longer in use.

Note that this is a general guideline pertaining to all Dell Compellent installations, and is not specific to Fluid Cache.

For more information and the procedure for removing volume information, refer to the section about removing volumes in Red Hat Enterprise Linux (RHEL) 6x Best Practices.

Fluid Cache for SAN Cluster Creation and Management Operations 27

Deleting a Volume From a Fluid Cache Cluster Use Enterprise Manager to completely delete a volume from a Fluid Cache cluster while maintaining just the cluster.

1 Click the Storage view.

2 In the Storage pane, expand Fluid Cache Clusters if necessary and select the cluster with the mapped volume to delete.

3 In the Cache tab, expand Volumes, select the volume to be deleted and click Delete. The Delete dialog box appears.

4 (Optional) Select Do you want to delete the Storage Center volume associated with the selected Fluid Cache volumes? to also delete the volume on the Storage Center.

CAUTION: If you select this option, it will delete the logical volume from the Storage Center and all the data contained in that volume will be permanently lost.

5 Click OK.

Adding Servers to a Fluid Cache Cluster To add servers to a Fluid Cache cluster:

1 Log in to Enterprise Manager as a user with Administrator or Volume Manager privileges.

2 In the Storage view, expand Fluid Cache Clusters if necessary.

3 Right-click the Fluid Cache cluster (not Fluid Cache Clusters itself) and click Tasks Add Servers to Cluster.

The Add Servers to Cluster dialog box is displayed. By default, all servers are selected that have Fluid Cache installed.

4 Clear the check boxes next to unwanted servers or select the Unselect All option, and then select the servers to be added. (To use all available servers again, select the Select All option.)

5 Click OK.

The system processes for some time, and then the server appears inside Fluid Cache Clusters in the Storage tab.

The Add Devices to Cluster dialog box is displayed. By default, all devices compatible with Fluid Cache are selected.

6 Clear the option next to unwanted cache devices or click the Unselect All option, and then select the cache devices to be added. (Select the Select All option to use all available cache devices again.)

CAUTION: Any existing data on a cache device is lost when the device is added to the Fluid Cache cluster. Back up this data before proceeding.

7 Click OK.

The device now appears in the list in the Devices section.

If you have a shared data application such as a cluster file system or clustered application, you may want to add the server to a server cluster (a subcluster) inside the Fluid Cache cluster. See Mapping Volumes. To create a subcluster, refer to the Enterprise Manager Administrators Guide.

Fluid Cache for SAN Cluster Creation and Management Operations 28

Removing a Server from a Fluid Cache Cluster 1 If the server belongs to a server cluster (a subcluster) within a Fluid Cache cluster, remove the server

from the subcluster:

a Prior to removing the server from a Fluid Cache cluster, you must shutdown the host or stop the Fluid Cache service.

b In Enterprise Managers Storage view, select the appropriate Storage Center. (Do not select Fluid Cache Clusters or its contents.)

c In the Storage tab, expand Servers if necessary and locate the server.

d Right-click the server and select Remove Server from Cluster. e When asked to confirm the action, click OK.

In the Storage tab, the server now appears outside of the subcluster, but is still inside the Fluid Cache cluster.

NOTE: Removing the server from the subcluster removes all mappings created for the subcluster.

2 Remove all Fluid Cache mappings from all volumes mapped to that server (see Removing Volume Mappings). Note that you do not have to remove non-Fluid Cache mappings.

3 Remove the server from the Fluid Cache cluster:

a In the Storage view, select the Fluid Cache cluster. (Do not select Storage Center or its contents.)

b In the Summary tab, locate the server in the Servers section.

c Right-click the server and select Remove Server from Cluster. d When asked to confirm the action, click OK.

After the system processes for some time, the server reappears outside of the cluster in the Servers area. You may need to manually refresh the display.

NOTE: If you exit Fluid Cache on a server and then remove the server from a cluster, you must run HCN Setup on the server to add the server to any cluster, including the cluster from which it was removed.

NOTE: To completely remove the Fluid Cache software from the server, run this command: rpm -e $(rpm -qa | grep FLDC). Although this command removes all configuration information, in

some cases you may also want to remove all components of Fluid Cache from your installation by deleting the installation directory: /opt/dell/fluidcache/

NOTE: Do not disable the Fluid Cache agent service running the chkconfig command.

Adding Cache Devices to a Fluid Cache Cluster 1 Log in to Enterprise Manager as a user with Administrator or Volume Manager privileges.

2 In the Storage view, expand Fluid Cache Clusters if necessary, and then right-click the Fluid Cache cluster. (Do not right-click Fluid Cache Clusters itself.)

3 Click Tasks Add Devices to Cluster.

The Add Devices to Cluster window is displayed. By default, all available devices compatible with Fluid Cache are selected.

4 Clear the option next to unwanted cache devices or click the Unselect All option, and then select the cache devices to be added. (Click the Select All option to use all available cache devices again.)

Fluid Cache for SAN Cluster Creation and Management Operations 29

CAUTION: Any existing data on a cache device is lost when the device is added to the Fluid Cache cluster. Back up this data before proceeding.

5 Click OK.

The devices now appear in the list in the Devices section.

Setting Storage Capacity for each Cache Server The maximum SSD storage capacity supported is 3.2 TB per node. However, the size of the cache devices on a fluid cache node affects the memory utilization of that server node. The number of nodes in the cluster also affect the memory utilization.

Therefore, if you want to use more than 1.6 TB of cache per node then a minimum memory configuration of 128 GB is recommended for each server node.

Removing a Cache Device from a Fluid Cache Cluster If you are removing the cache device from the server, perform a graceful removal by first ensuring that the cache device is not in active use. See Removing a Server from a Fluid Cache Cluster for the steps on gracefully removing a server from a cluster.

1 In Enterprise Managers Storage view, expand Fluid Cache Clusters if necessary, and then select the Fluid Cache cluster. (Do not select Storage Centers or its contents.)

2 In the Summary tab, locate the cache device in the Devices section.

3 Right-click the device and select Remove Device from Cluster.

4 When asked to confirm the action, click OK.

As part of the deletion process, dirty data in the cache is flushed to main storage. This could take a considerable duration of time for a large quantity of data.

When processing is complete, the cache device no longer appears in the list of devices. You may need to manually refresh the data on the page.

Reactivate a Cache Device Attached to a Fluid Cache Cluster Use Enterprise Manager to reactivate a Fluid Cache cluster cache device.

1 Click the Storage view.

2 In the Storage pane, expand Fluid Cache Clusters if necessary and select the Fluid Cache cluster.

3 In the Cache or Summary tab, expand Servers if necessary, and expand the server housing the device if necessary. Select the device to be reactivated and click Reactivate Device. The Reactivate Device dialog box appears.

4 Click OK.

Fluid Cache for SAN Cluster Creation and Management Operations 30

Reactivate a Volume on a Fluid Cache Cluster Use Enterprise Manager to reconnect to a volume in a failed state from a Fluid Cache cluster while maintaining the cluster and the volume.

1 Click the Storage view.

2 In the Storage pane, expand Fluid Cache Clusters if necessary and select the cluster with the questionable volume.

3 In the Volumes pane of the Summary tab, double-click the volume to be reactivated and click Reactivate Volume. The Reactivate Volume dialog box appears.

4 Click OK.

Adding a Storage Center to a Fluid Cache Cluster 1 Log in to Enterprise Manager as a user with Administrator or Volume Manager privileges.

2 Click the Storage view.

3 In the Storage pane, expand Fluid Cache Clusters if necessary, and then select the Fluid Cache cluster.

4 In the Cache tab, select Tasks and click Assign Storage Centers.

The Assign Storage Centers window is displayed.

5 In the Assign Storage Centers window, select one or more Storage Centers to be added.

6 Click OK.

Remove a Storage Center from a Fluid Cache cluster Use Enterprise Manager to remove a Storage Center from a Fluid Cache cluster while keeping the cluster.

1 Click the Storage view.

2 In the Storage pane, expand Fluid Cache Clusters if necessary and select the Fluid Cache cluster.

3 In the Cache or Summary tab, expand Storage Centers if necessary, select the Storage Center to be removed and click Remove Storage Center from Cluster. The Remove Storage Center from Cluster dialog box appears.

4 Click OK.

NOTE: The Storage Center cannot be removed if the cluster has volumes mapped to it.

Fluid Cache for SAN Cluster Creation and Management Operations 31

Reconnect a Fluid Cache Cluster to a Storage Center Use Enterprise Manager to reconnect a Storage Center to a Fluid Cache.

1 Click the Storage view.

2 In the Storage pane, expand Fluid Cache Clusters if necessary and select the Fluid Cache cluster.

3 In the Cache tab, select Storage Centers and click Reconnect Host Cache Cluster to Storage Center.

Change the License for a Fluid Cache cluster Use Enterprise Manager to change the license for a Fluid Cache cluster.

1 Click the Storage view.

2 In the Storage pane, expand Fluid Cache Clusters if necessary and select the Fluid Cache cluster.

3 Click Edit Settings. The Edit Settings dialog box appears.

4 Click Submit License. The Submit License dialog box appears.

5 Click Browse. Navigate to the location of the new license file, select it, and click Save.

6 Click OK.

Send Fluid Cache Cluster Information Using Dell SupportAssist Use Enterprise Manager to send Fluid Cache cluster information to technical support using Dell SupportAssist.

1 Click the Storage view.

2 In the Storage pane, expand Fluid Cache Clusters if necessary and select the Fluid Cache cluster.

3 Click Edit Settings. The Edit Settings dialog box appears.

4 Click Dell SupportAssist, and then click Send Dell SupportAssist Data Now. The Send Dell SupportAssist Now dialog box appears.

5 Click OK.

Shutting Down a Fluid Cache Cluster Use Enterprise Manager to shut down a Fluid Cache cluster.

1 Click the Storage view.

2 In the Storage pane, expand Fluid Cache Clusters if necessary and select the Fluid Cache cluster.

3 Click Shutdown. The Shutdown dialog box appears.

4 Click Yes.

Fluid Cache for SAN Cluster Creation and Management Operations 32

CAUTION: All cached volumes and their data become inaccessible when the Fluid Cache cluster is shut down, unless they're remapped to another cluster first.

If you do not remap the volumes before shutting down, call Dell Technical Support Services for help remapping a shut down cluster.

NOTE: Volumes mapped to a Fluid Cache cluster that's been shut down can't be remapped anywhere other than a Fluid Cache cluster.

Enable Server Load Equalizing for Storage Center Volumes Server load equalizing dynamically adjusts queue depth for volumes experiencing high IOPS to minimize the performance impact on other volumes. Enable load equalizing on a Storage Center that hosts Fluid Cache volumes to prevent cache flushing operations from adversely affecting performance for other volumes.

NOTE: Enable load equalizing only for environments using Fluid Cache clusters, or if directed by Dell Technical Support Services.

1 Click the Storage view.

2 In the Storage pane, select the Storage Center.

3 In the Summary tab, click Edit Settings. The Edit Settings dialog box appears.

4 Click the Storage tab.

5 Select the Server Load Equalizer Enabled check box.

6 Click OK.

Maintaining Fluid Cache Environments This chapter describes tasks that must occasionally be performed after initial Fluid Cache deployment.

WARNING: When performing any operation to a server platform that hosts Fluid Cache software, you must perform either one of the following methods:

Shut down a single Fluid Cache Cluster node at a time and perform the host-based server updates.

After the updates to host server are complete, reboot the host and bring the Fluid Cache Cluster node online. Before shutting down and updating the next Fluid Cache cluster node, verify that the Fluid Cache Cluster node is added to Fluid Cache Cluster appropriately.

Place the entire Fluid Cache Cluster into maintenance mode, and then shut down the entire Fluid Cache Cluster by shutting down the Fluid Cache Cluster nodes one after the other.

After the updates to the host servers are complete, reboot the server and bring the Fluid Cache Cluster nodes online.

WARNING: To avoid potential performance issues and data loss, you must NOT shutdown multiple Fluid Cache Cluster nodes at the same time to perform Fluid Cache host server based updates.

Fluid Cache for SAN Cluster Creation and Management Operations 33

Shutting Down and Restarting a Cluster Shut down a cluster if, for example, you need to perform system maintenance but do not need to make any configuration changes to the cache network itself.

To shut down a cluster:

1 Exit any applications that access cached volumes.

2 In Enterprise Manager, click Storage.

3 In the Storage pane, expand Fluid Cache Clusters if necessary, and then select the Fluid Cache cluster.

4 Click Shutdown.

The Shutdown window is displayed.

5 Click Yes.

The system processes for some time while data is flushed. There may be a significant delay for large amounts of data.

NOTE: While a cluster is shut down, all cached volumes and their data are inaccessible. If you need to maintain access to these volumes while the cluster is shut down, remap the volumes before shutting down the cluster.

To restart a cluster:

1 Make sure the SAN and Storage Center are fully operational, and the cache network is configured and running.

2 In Enterprise Manager, click the Storage view.

3 In the Storage pane, expand Fluid Cache Clusters if necessary and select the Fluid Cache cluster.

4 Click Restart.

Uninstalling the Fluid Cache Software Uninstall the Fluid Cache software if a server will no longer be used for Fluid Cache.

Before uninstalling the software, you must remove all volume mappings and remove the server from its Fluid Cache cluster. See Removing Volume Mappings and Removing a Server from a Fluid Cache Cluster.

To uninstall the Fluid Cache software, run the command:

rpm -e $(rpm qa | grep FLDC)

Although this command removes all configuration information, in some cases you may want to remove all components of Fluid Cache from your installation by also deleting the installation directory: /opt/dell/ fluidcache/

Fluid Cache for SAN Cluster Creation and Management Operations 34

Fluid Cache Web Page Overview The Fluid Cache web page collects all possible information about the Fluid Cache clusters and provides detailed status and activity report about the hosts and cache devices within the clusters.

Topics:

Accessing Fluid Cache Web Page

Understanding Fluid Cache Web Page

Accessing Fluid Cache Web Page In the address bar, enter the URL of one of the hosts that is part of Fluid Cache cluster on port 8082. Use the following format:

http:// :8082

The browser will display the Fluid Cache web page.

Understanding Fluid Cache Web Page This section describes the Overview page of the Fluid Cache Web page.

Status The status section of the Overview page displays the following information.

Version Displays the version and build number of Fluid Cache cluster.

Start Time Displays the time that the cluster was started.

Last Update Displays the time that the cluster was last modified.

Cluster State Displays the state of the cluster.

CFM Info The CFM Info section displays the following information about the primary (active) CFM in the cluster.

6

Fluid Cache Web Page Overview 35

Peering state Displays the role of the CFM, The possible options are Primary or Secondary. You can have only one primary and two secondary CFMs up at any time.

Hostname Displays the hostname of this node.

Address Displays the management IP address of this node.

Listen Interfaces Displays the cache network IP address of this node.

Peer CFMs The Peer CFMs section displays the following information about other Fluid Cache hosts that serves as CFMs.

Hostname Displays the hostname of the Secondary CFMs in this cluster.

Address Displays the cache network IP addresses of the Secondary CFMs.

Port Displays the TCP ports that are used to communicate between the CFMs.

Online Displays the connection state of the Secondary CFMs. The possible options are Yes or No.

Metadata Server Stats The Metadata Server Stats section displays the following information about the Fluid Cache cluster hosts serving as Metadata Servers.

Hostname Displays the hostname of the three Metadata servers in the cluster.

Status Displays the status of the Metadata servers. The possible options are online or offline.

Connection Displays the connection protocol used for the Metadata servers. The connection protocols should be RDMA/RC.

Active Displays whether the Metadata server is Active. The possible options are Yes or No

Requests Displays the number of Metadata requests to each Metadata Server.

Cache Server Stats The Cache Server Stats section displays the following information about a single Fluid Cache Cluster host.

Hostname Displays the hostname of the host.

Connection Displays the connection method or protocol used by the host to communicate with other hosts in the cluster.

Devices Displays the details about the cache devices present in the host.

In Use (GB) Displays the cache memory space (in GB) that is being used on the host. This include data stored in replica blocks.

In Use (%) Displays the percent of cache memory space that is being used on the host.

Client Reading (GB) Displays the cumulative data (in GB) that is being read by a client.

Client Writing (GB) Displays the cumulative data (in GB) that is being written by a client.

Dirty (GB) Displays the number of unflushed blocks that exist on the cache server on this host.

Evicted (GB) Displays the cumulative data (in GB) that was evicted from this host.

Fluid Cache Web Page Overview 36

Fill Read Ops Displays the cumulative number of cold read operations by the cache server on this host

Failed Fill Read Ops Displays the number of cold read operations that failed.

Slow Fill Read Ops Displays the number of cold read operations that took longer than two seconds.

Failed Flush Write Ops- Displays the number of write operations on Compellent that failed.

Slow Flush Write Ops Displays the number of write operations on the Compellent that took longer than two seconds.

Fill read (GB) - Displays the date (in GB) that have been read by the host from the Compellent.

Fill read rate (GB) Displays the rate (in GB) at which host data is being read from the Compellent.

Flush write (GB) Displays the rate ( in GB) at which data is being written on the Compellent from the host.

Flush write rate (MB/s) Displays the current rate at which the host is writing on the Compellent. This is usually limited by the Compellent.

Flush write dirty (GB) Displays the data (in GB), which is dirty, was written on the Compellent.

Flush write dirty rate (MB/s) Displays the current rate at which the host is writing dirty blocks on the Compellent. This is usually limited by the Compellent.

Write Same (GB) Not supported.

Write Same rate (MB/s) Not supported.

Cache Server Per-Path Stats The Cache Server Per-Path Stats displays the information about the activity of hosts based on each cached LUN.

The first row displays the LUN ID of the cached LUN and the total activity the hosts mapped to the LUN. The subsequent rows displays the storage paths mapped for the hosts in the LUN and about the activity of each node.

The following information is displayed in the Cache Server Per-Path Stats section and the row shows the value for each cache server relative to each volume

Storage Path Displays the storage path mapped from the Compellent to the Fluid Cache VSA.

Cache Server Displays the hostname of the cache server.

Availability Displays if the particular cache server is up and running. The possible options are Yes or No.

Cached (GB) Displays the amount of data that has been cached.

Replica (GB) Displays the number of replica bytes that exists.

Dirty (GB) Displays the amount of dirty data that exists.

Fill read (GB) Displays the total amount of data that has been read from the Compellent...

Fill read rate (MB/s) Displays the current rate at which data is being read from the Compellent in MB/s.

Flush write (GB) Displays the total amount of data that has been written to the Compellent.

Fluid Cache Web Page Overview 37

Flush write dirty (GB) Display the total amount of dirty data that has been written to the Compellent, may be different from the Flush write because whole cache blocks can be written to the Compellent even if all of the sectors within the cache block are not dirty.

Reads/s Display the current rate of reads per second from the Compellent.

Writes/s Display the current rate of writes per second to the Compellent.

Flush write rate (MB/s) Display the current rate at which data is being written to the Compellent in MB/s.

Flush write dirty rate (MB/s) Display the current rate at which dirty data is being written to the Compellent in MB/s.

Write Same (GB) Not Used.

Write Same rate (MB/s) Not Used.

Block Device Stats The Block Device Stats section displays the activity of each Fluid Cache device. The first row displays the internal device name of the cached volume and the total activity of the hosts accessing the cache volume. The rows displays the activity of individual nodes that are accessing the cache volume. A normal, healthy cluster displays a status of online for each nodes access to the Fluid Cache device.

Device Displays the name of the Fluid Cache block device.

Hostname Displays the hostname of the nodes that have that block device mapped to them.

Connection Displays the connection protocol the block device uses to connect to the cache servers. The connection protocols should be RDMA/RC.

Size(GB) Displays the size of the Compellent Volume that is being cached in GB.

Persistent Displays the WWN of the Compellent backing store, note the 8 in the middle of all the zeros of the WWN.

Status Displays the current status of the vblock device on each node. If the device status is offline, the issue that took the device offline must be resolved, and then the device must be reactivated in EM.

Openers Displays the number of processes that have the device open.

Read (GB) Displays the number of GB that have been read from the block device.

Write (GB) Displays the number of GB that have been written to the block device.

Read rate (MB/s) Displays the Current rate at which data is being read from the block device in MB/s.

Write rate (MB/s) Displays the current rate at which data is being written to the block device in MB/s

Reads/s Displays the current number of reads per seconds that are being read from the block device.

Write/s Displays the current number of writes per second that are being written to the block device.

Fluid Cache Web Page Overview 38

Troubleshooting Fluid Cache Installations

If you have issues running Fluid Cache after a successful completion of the installation procedure, contact your Compellent Copilot.

Troubleshooting the Compellent array and SAN architecture is beyond the scope of this document.

For additional troubleshooting information, refer to the Enterprise Manager Administrators Guide and the documentation for other hardware and software components. See Related Documentation.

After the initial list of Basic Troubleshooting Steps, the troubleshooting topics are presented in the order in which the issues are likely to appear as you deploy, configure, administer, and maintain a Fluid Cache installation.

Topics:

Basic Troubleshooting Steps

Cannot Set up Fluid Cache on a Server

Unable to remove a cache device from Fluid Cache Cluster using Enterprise Manager

Incorrect Network Address or Netmask

Server Does Not Appear in List of Servers

Cache Device Does Not Appear in List of Cache Devices

Cache Device Cannot Be Added to a Cluster

Cannot Select a Specific Cache Mode

Fluid Cache Node Is Unavailable in Enterprise Manager

Cached LUNs Are Unavailable

Cannot Create a Fluid Cache Cluster

Fluid Cache License Is Expired

Fluid Cache License Is Invalid

Cannot Configure the Cache Network

Cannot Assign or Remove a Storage Center

Cannot Determine Which Cache Device Failed

Events for Fluid Cache Are Not Shown in Enterprise Manager

Cluster or Application Has Performance Issues

Unable to Recreate a Fluid Cache Cluster After a Hardware Failure

Cache Node is Not Listed

Unable to Select a Specific Caching Mode

Fluid Cache License File is Invalid

Option to Create Cluster Not Available

A

Troubleshooting Fluid Cache Installations 39

Unable to Add a Volume to a Fluid Cache Cluster

Event Messages Are Not Being Delivered

Storage Center is Not Available

Fluid Cache Server is Not Available

Information Displays Differently Between Storage Centers and Fluid Cache Clusters

Verify That All Parts of the Fluid Cache Cluster are Communicating with Each Other

Verify the Data Path is Working

The Cluster in the Fluid Cache Clusters Display is Marked Red

Problems Configuring Server Clusters Defined on a Storage Center with Dell Fluid Cache for SAN

Basic Troubleshooting Steps Make sure the following conditions are fulfilled:

You have downloaded and installed the most recent firmware, drivers, and software required to support Fluid Cache for SAN (see Requirements for Fluid Cache).

The hardware is racked and cabled according to your hardware documentation.

The network security settings match those specified in Checking Network Connections.

The ports available to Fluid Cache match those specified in Fluid Cache for SAN Environment.

All servers are in the same management interface subnet and in the same cache network subnet.

Each server in the Fluid Cache cluster appears in Enterprise Managers list of servers.

The avahi-daemon is running.

(Optionalfor systems with multipath enabled and running) The Fluid Cache devices have been excluded in the /etc/multipath.conf file on each server in the cluster. See Checking Network Connections.

Cannot Set up Fluid Cache on a Server Possible Cause HCN Setup could not set up Fluid Cache on a server because one of the Linux

dependencies is not installed.

Solution Follow the instructions given by RPM, which should state the dependencies that are missing. Check for the dependencies listed in the Release Notes and compare them with the list of currently installed dependencies, which you can check by running this command: rpm -qa Fluid Cache uses Avahi for autodiscovery between cluster nodes. To check that the Avahi daemon is running on a node, run the command: ps ef | grep avahi- daemon. To start the Avahi daemon, run the command: /etc/init.d/avahi-daemon start. To ensure that the Avahi daemon starts when the server is restarted, run the

command: chkconfig avahi-daemon on. If your avahi-daemon.conf file does

not use the default settings, make sure that Disable Publishing is set to no, that Deny Interfaces does not list Fluid Cache interfaces, and that Allow Interfaces is either commented out or lists all Fluid Cache interfaces.

Troubleshooting Fluid Cache Installations 40

Possible Cause HCN Setup could not set up Fluid Cache on the server because the MPIO service was configured for multipathing, but devices required by Fluid Cache were not blacklisted in the etc/multipath.conf file.

Solution Add blacklist entries for the devices required by Fluid Cache. On each node in the cluster, modify the Devices section of the /etc/multipath.conf file as follows:

blacklist { devnode "^fldc[0-9]*" devnode "^nvme[a-z]*" devnode "^rssd[a-z]*" }

After creating the entries and saving the file, start Fluid Cache by running this command: service fldc_agentd start

Unable to remove a cache device from Fluid Cache Cluster using Enterprise Manager Description In Enterprise Manager, when you perform operations such as removing a cache device

from a Fluid Cluster node, the following error message is displayed:

Fluid Cache not Found

Possible Cause Enterprise Manager may not be in synchronization with the Fluid Cache cluster to retrieve the latest state.

Solution To resolve this issue, shutdown the EM client, and then restart the EM Data Collector.

If the issue persists, contact Dell Customer Support.

Incorrect Network Address or Netmask Possible Cause While setting up a server, incorrect values were entered for the network address or

netmask of either the management interface or the cache network.

Solution Restart HCN Setup and enter the correct values. See Setting up the Fluid Cache Servers.

Troubleshooting Fluid Cache Installations 41

Server Does Not Appear in List of Servers Possible Cause A configuration issue is preventing the server from appearing in the list.

Solution From the server, run the command ip addr. The cache network interfaces state

should display as UP. If not, recheck the server configuration. See

Preparing the Fluid Cache Servers and Setting up the Fluid Cache Servers.

Possible Cause Firewall or iptables settings are preventing network communication.

Solution Check your firewall and iptables settings. See Checking Security Settings.

Possible Cause The network switch is not correctly cabled or configured.

Solution Review the settings for the network switch and consult your switch documentation. See Cluster or Application Has Performance Issues.

Possible Cause The Fluid Cache agent is not running on the server.

Solution From the server, run the command: service fldc_agentd start Possible Cause The Avahi daemon is not running on the server.

Solution From the server, run the command: service avahi-daemon start.

Cache Device Does Not Appear in List of Cache Devices Possible Cause The cache device you are trying to add is not supported by Fluid Cache.

Solution Refer to the list of supported devices in Requirements for Fluid Cache.

Possible Cause The driver or firmware for the cache device is not a supported version.

Solution Make sure that your cache device firmware and driver are up to date and that the driver is loaded.

Firmware: Run the command: dmesg | grep mtip32xx | grep Firmware

In the output, the firmware must version displayed must be B1490508 or later.

Driver: Run the command: dmesg | grep -i micron

In the output, the driver version for the device named Micron RealSSD PCIe must be 2.4.2 or later.

Possible Cause The cache device is not functioning properly.

Troubleshooting Fluid Cache Installations 42

Solution To check device function, select the device in Enterprise Manager and in the Event tab, look for a device failure message. Replace the cache device if necessary, using instructions in the Dell Compellent Enterprise Manager Users Guide.

Cache Device Cannot Be Added to a Cluster Possible Cause The cache device is not functioning properly. Under some conditions, the process of

adding a device completes normally even though the device being added is not functioning properly.

Solution To check device function, select the device in Enterprise Manager and in the Event tab, look for a device failure message.

If necessary, replace the cache device by following the instructions provided in the Dell Compellent Enterprise Manager Users Guide.

Cannot Select a Specific Cache Mode Possible Cause There is an existing cache mode configured for the volume mapping. The cache mode

chosen for a volume mapping cannot be changed. The existing mapping must be deleted, and a new mapping created. A new cache mode can be selected while creating the new mapping.

Solution Remove the volume mapping and create a new mapping. When you create the new mapping, you can then select a different cache mode (cache policy). See Removing Volume Mappings and Mapping Volumes.

Possible Cause The cluster is in maintenance mode.

Solution Take the Fluid Cache cluster out of maintenance mode by selecting the cluster in Enterprise Manager, clicking Edit Settings in the Summary tab, and clearing the Maintenance Mode option. Note that certain system failures or an invalid license may prevent the cluster from being taken out of maintenance mode. Refer to the Enterprise Manager Status and Events tabs, and see Fluid Cache License Is Expired.

Troubleshooting Fluid Cache Installations 43

Fluid Cache Node Is Unavailable in Enterprise Manager Possible Cause The node failed during restart. The node was unable to perform a graceful shutdown

because the chkconfig command was used to shut down the Fluid Cache agent

service.

Solution Do not use the chkconfig command to disable the Fluid Cache agent service.

Perform normal node recovery procedures to return the node to normal operation.

Cached LUNs Are Unavailable Possible Cause After restarting Fluid Cache or restarting a server, cached LUNs may not be immediately

available.

Solution Wait for the cached LUNs to reappear. It may take a significant amount of time to write a large amount of data. The cached volumes reappear after the cache completes recovery.

Possible Cause A fault in the cluster is preventing the volume from being recognized by Enterprise Manager.

Solution Check the Enterprise Manager status page and event log for error messages.

Cannot Create a Fluid Cache Cluster Possible Cause A minimum of three PowerEdge servers have not been configured for use with Fluid

Cache. Until at least three servers have Fluid Cache installed and configured, the option to create a Fluid Cache cluster is unavailable.

Solution Add three or more nodes and configure the nodes before attempting to create a Fluid Cache cluster in Enterprise Manager.

Possible Cause Network connectivity issues are preventing the creation of a Fluid Cache cluster.

Solution Check that the security settings are correct and the required ports and network connections are open. See Checking Network Connections and Checking Security Settings.

Troubleshooting Fluid Cache Installations 44

Fluid Cache License Is Expired Possible Cause System settings such as changes to the system date cause the current Fluid Cache

license to expire. You can still access data on cached volumes, but performance is degraded because the Fluid Cache cluster has been placed in maintenance mode and caching is no longer active.

Solution Check the status of the license file by selecting the Fluid Cache cluster in Enterprise Manager and referring to the status shown on the Events or Cache tabs. If the license is expired, make sure that your system settings for date and time are correct. If it is not correct, set the appropriate date and time. Take the Fluid Cache cluster out of maintenance mode by selecting the cluster in Enterprise Manager, clicking Edit Settings in the Summary tab, and then clearing the Maintenance Mode option.

Possible Cause Fluid Cache for SAN is running on an evaluation license (typically 90 days) and that time period has been exceeded. You can still access data on cached volumes, but performance is degraded because the Fluid Cache cluster has been placed in maintenance mode and caching is no longer active.

Solution Check the status of the license file by selecting the Fluid Cache cluster in Enterprise Manager and referring to the license type shown on the Events or Cache tabs. If the number of days remaining is zero, contact your Dell representative to purchase a Fluid Cache for SAN license. After activating the new license, take the Fluid Cache cluster out of maintenance mode by selecting the cluster in Enterprise Manager, clicking Edit Settings in the Summary tab, and deselecting Maintenance Mode.

Fluid Cache License Is Invalid Possible Cause The license file is invalid if it is in any way modified. This causes unsuccessful digital

signature validation.

Solution Contact Dell Customer Support.

Cannot Configure the Cache Network Possible Cause Firewall settings or IP table entries are preventing access to one or more ports required

by Fluid Cache.

Solution Change your firewall settings to allow access by Fluid Cache. For a list of required ports, see Checking Network Connections. To check which ports are currently in use, log in as a root user and run the command:

/bin/netstat -tulpn

Troubleshooting Fluid Cache Installations 45

Also, check for iptables entries that may be blocking Fluid Cache network traffic. Note that some default installations for RHEL create an iptables entry for ib_send_bw that

prevents connections to another server and thus blocks Fluid Cache network traffic.

Possible Cause One of the ports required by Fluid Cache is in use by another process.

Solution Refer to the required ports listed in Checking Network Connections and reassign ports as needed.

Check which ports are in use by logging in as a root user and running this command:

/bin/netstat -tulpn To change the ports used by Fluid Cache, open the /etc/services file on each

server and add the following information, modifying the port numbers as needed:

fldc-http 8080/tcp # Fluid Cache CFM Web Server fldc-https 6773/tcp # Fluid Cache CFM Web Server fldca-http 8082/tcp # Fluid Cache Agent Web Server fldca-https 6774/tcp # Fluid Cache Agent Web Server The existing entry for the mDNS service on UDP port 5353 may also need to be modified. It is used by the Avahi daemon, which is required by Fluid Cache.

After changing the ports, restart the Fluid Cache agent service by running the following command:

service fldc_agentd restart

NOTE: Port 6774 is used to communicate with Enterprise Manager. If you change the port number for fldca-https in the etc/services file, you must also

configure Enterprise Manager to use the new port number.

NOTE: Port 3033 is used to communicate with Enterprise Manager and Storage Center. To configure those applications to use a different port, refer to the corresponding Administrators Guide. After a new port is configured in Enterprise Manager or Storage Center, Fluid Cache automatically uses the new port.

NOTE: Port 7449 is required by Fluid Cache. If another application is using this port, configure that application to use a different port.

Possible Cause The network adapters are not configured for Ethernet functionality, and so the ethtool and ifconfig commands cannot be used to configure the network

adapters.

Solution On each node in the Fluid Cache cluster, run the command: connectx_port_config In the configuration file, if the entry for ConnectX PCI Devices references auto or ib, change the setting for Select mode for port x to 2.

Troubleshooting Fluid Cache Installations 46

Cannot Assign or Remove a Storage Center Possible Cause The Storage Center is already assigned to another Fluid Cache cluster.

Solution In Enterprise Manager, see whether or not a Storage Center is listed for the Fluid Cache cluster.

Possible Cause Network connectivity issues are preventing Enterprise Manager from communicating with Storage Center.

Solution Make sure the network is functioning properly. Refer to Checking Network Connections, Checking Security Settings.

Cannot Determine Which Cache Device Failed Possible Cause One of the cache devices has failed on a server with multiple cache devices installed,

and it is not clear which of them failed.

Solution In Enterprise Manager, in the Cache tab, the failed cache device is identified by a red X through it. Note the last digits of the number for this cache device. This number matches the serial number printed on the label of the failed cache device.

Events for Fluid Cache Are Not Shown in Enterprise Manager Possible Cause When Enterprise Manager was installed, the Data Collector was not configured for the

correct IP address.

Solution Make sure the IP address that the Data Collector uses is accessible by the Fluid Cache nodes. To view the IP address, start the Data Collector Manager, select General Information, and note the IP address within the URL in the Web Site field. Change the address if necessary and restart the Data Collector.

Troubleshooting Fluid Cache Installations 47

Cluster or Application Has Performance Issues Possible Cause One or more cache devices are uninstalled, have failed, or do not have the correct

firmware or drivers.

Solution Use Enterprise Manager to check the functionality of the cache devices.

Possible Cause The Compellent storage array is overloaded.

Solution In Enterprise Manager, check the storage latencies and throughput on the cached volumes. Add more capacity to the Compellent array if necessary.

Possible Cause The application is not making use of the cache.

Solution Make sure the application has an I/O profile that can leverage Fluid Cache, which accelerates reads, writes, and read-after writes. If the application does, check the running cache mode for that volume in Enterprise Manager. If necessary, change cache modes by deleting the volume mapping and creating a new one with a different cache mode. See Mapping Volumes.

Possible Cause The active data set greatly exceeds the size of the cache pool.

Solution Add more cache devices to increase the size of the cache pool.

Possible Cause Network traffic from outside Fluid Cache is interfering with performance.

Solution The cache network switch is not configured correctly. Configure the ports on the switch used by Fluid Cache so that they are used solely by Fluid Cache.

Possible Cause The cache network switch is not configured correctly.

Solution Make sure that the switch has the following settings:

The ports are in Layer 2 mode.

The ports are in an untagged state.

The switch firmware is up to date.

Multicast is enabled.

Verify that flow control is enabled and DCB is disabled. For Dell Networking switches, see Configuring the Cache Network Switch. For all other switches, consult the manufacturers documentation.

NOTE: Enabling flow control is a requirement for Fluid Cache.

For blade enclosure, disable FlexAddress.

You can check network functionality by checking rx_over_errors using the

ethtool -S command.

Possible Cause Fluid Cache is running on an evaluation license (typically 90 days) and that time period has been exceeded. You can still access data on cached volumes, but performance is

Troubleshooting Fluid Cache Installations 48

degraded because the Fluid Cache cluster has been placed in maintenance mode and caching is no longer active.

Solution Check the status of the license file by selecting the Fluid Cache cluster in Enterprise Manager and referring to the status shown on the Events or Cache tabs. Contact your Dell representative to purchase a Fluid Cache license. After activating the new license, take the Fluid Cache cluster out of maintenance mode by selecting the cluster in Enterprise Manager, clicking Edit Settings in the Summary tab, and then clearing the Maintenance Mode option.

Unable to Recreate a Fluid Cache Cluster After a Hardware Failure Use Enterprise Manager to completely delete a Fluid Cache cluster that has failed before trying to remake the cluster.

NOTE: Clean up Enterprise Manager and the Storage Centers after a hardware failure before recreating a Fluid Cache cluster. The Delete option will not appear unless the Fluid Cache cluster has been fully removed from Enterprise Manager.

Cache Node is Not Listed The node was not selected in the Select Servers window when creating the Fluid Cache cluster or the network switch may not be configured correctly.

1 Make sure the node is selected.

2 If the node is still not listed, review the settings for the network switch and consult your switch documentation.

Unable to Select a Specific Caching Mode The required number of nodes or PCIe SSDs may not be available.

Keep in mind that while a cache device or cache server may have been previously configured, it may have since failed or is impacted by a network failure.

For write-back caching, ensure that there is a minimum of two nodes with PCIe SSDs on different servers in the Fluid Cache cluster.

For write-through caching, ensure that there is at least one node with a PCIe SSD.

Ensure the cluster is not in maintenance mode.

Troubleshooting Fluid Cache Installations 49

Fluid Cache License File is Invalid Verify that the license didnt expire or that a system change caused the license to be invalidated.

The Fluid Cache license status can be verified on either the Fluid Cache clusters Events tab or Cache tab.

An evaluation license is valid for only 90 days. Contact your Dell sales representative to purchase a Dell Fluid Cache for SAN license.

Option to Create Cluster Not Available All of the prerequisites for creating a Fluid Cache Cluster must be satisfied before the option to do so appears.

Must be running Enterprise Manager 6.5.1 or later.

Must have a Storage Center attached that can support Fluid Cache.

Must have a minimum of one Fluid Cache server configured for use with Fluid Cache.

Must be running Dell Fluid Cache for SAN 2.0 later.

Unable to Add a Volume to a Fluid Cache Cluster To verify the issue, check to see that both of the following items are true.

Make sure all the Fluid Cache servers are on the same network as the Storage Center that is serving the volume.

Make sure that the volume is not in use on another server or cluster.

Event Messages Are Not Being Delivered The Enterprise Manager Data Collector is responsible for receiving alert messages and transmitting them to configured users. When setting up Enterprise Manager on a multi-homed system, make sure to set messaging to use a NIC that is on the same network as the Fluid Cache server(s), or configure routing on the cache server to be able to get to the address on the NIC the Enterprise Manager Data Collector was configured to use.

Troubleshooting Fluid Cache Installations 50

Storage Center is Not Available If you receive an error that you dont have a Storage Center or you dont see any Storage Centers, make sure that the Storage Center(s) is running version 6.5.1 or later.

Fluid Cache Server is Not Available If you do not see a Fluid Cache server that you expect to be listed, click the Rescan button as the server may not have been discovered by the other servers in the cluster.

Information Displays Differently Between Storage Centers and Fluid Cache Clusters The Fluid Cache Clusters display shows information from the Fluid Cache server perspective while the Storage Centers display shows information from the Storage Center perspective. If there is a communication issue between the Fluid Cache servers and the Storage Center, they may each show that condition differently.

There is some latency in gathering information from the Fluid Cache servers by both Data Collector and the Storage Center as well as from the Dell Storage Client gathering information from Data Collector. It could take several minutes for all parts of the system to synchronize and show a consistent view.

Verify That All Parts of the Fluid Cache Cluster are Communicating with Each Other If the Dell Storage Client displays a "reconnect" message in the Fluid Cache display for a Fluid Cache cluster it means that EM is no longer able to communicate with the Fluid Cache cluster through the management network.

Verify that the network is operational between the Data Collector server and the Fluid Cache servers using a network tool such as ping.

If ping works but a reconnect to that address continues to fail, make sure that the Fluid Cache software is operational on the server.

Troubleshooting Fluid Cache Installations 51

If the EM client displays a red mark over the cluster in the Storage Centers view of the cluster, it means that the Storage Center is reporting that it cannot communicate with the cluster servers over the management network.

Verify that the network is operational between the cluster servers and the Storage Center by using a network tool such as ping.

Note that it may take several minutes for the Storage Center to report the cluster status (down or up).

Verify the Data Path is Working The Storage Centers server view is present if the Storage Center is able to see data connections to the cluster. The HBAs will be have a red mark if there are problems.

The Fluid Cache Clusters server view will show the HBAs from the Fluid Cache Servers perspective. Depending on the server and its operating system, conditions such as connectivity may not be visible from this view.

When checking for data connectivity, always check the Storage Centers view first.

The Cluster in the Fluid Cache Clusters Display is Marked Red Take the following into consideration when troubleshooting this problem.

Make sure the cluster is not in Maintenance mode.

Enterprise Manager will give the cluster a red mark if EM is having trouble communicating with the Fluid Cache servers or if the Fluid Cache servers report problems back to EM. The Cache tab for the cluster in the Fluid Cache Clusters display may have some text to indicate the issue being experienced. Also check the events for clues as to what is causing the issue.

Expand the tabs for each of the servers, volumes, and storage centers for the cluster in the Fluid Cache Clusters display. Additional red marks on a server, cache device, volume, or Storage Center may help determine why the cluster has a red mark.

Problems Configuring Server Clusters Defined on a Storage Center with Dell Fluid Cache for SAN When a Storage Center is assigned to a Fluid Cache cluster, the Fluid Cache cluster has to follow the same requirements as normal server clusters defined on the Storage Center. Servers in the Fluid Cache cluster are automatically mapped to an existing Storage Center server through their HBAs or created anew if no previous Storage Center server exists.

Troubleshooting Fluid Cache Installations 52

Since server clusters defined on a Storage Center must have the same operating system, all servers in the Fluid Cache cluster must also have the same operating system. If a Storage Center server with matching HBAs was previously created on the Storage Center prior to the assignment and it was defined with a different operating system on the Storage Center, than the Fluid Cache cluster will not be able to assign the Storage Center to the Fluid Cache cluster.

When a Fluid Cache cluster has servers that are part of a Storage Center server cluster, the Storage Center server cluster becomes a sub-cluster under the Fluid Cache cluster when the Storage Center gets assigned to the Fluid Cache cluster. To maintain cluster rules, if a server in the Fluid Cache clus

Manualsnet FAQs

If you want to find out how the Fluid Cache Dell works, you can view and download the Dell Fluid Cache SAN 2.1.0V2.1.0 Storage Deployment and Admin Guide For Linux Systems on the Manualsnet website.

Yes, we have the Deployment and Admin Guide For Linux Systems for Dell Fluid Cache as well as other Dell manuals. All you need to do is to use our search bar and find the user manual that you are looking for.

The Deployment and Admin Guide For Linux Systems should include all the details that are needed to use a Dell Fluid Cache. Full manuals and user guide PDFs can be downloaded from Manualsnet.com.

The best way to navigate the Dell Fluid Cache SAN 2.1.0V2.1.0 Storage Deployment and Admin Guide For Linux Systems is by checking the Table of Contents at the top of the page where available. This allows you to navigate a manual by jumping to the section you are looking for.

This Dell Fluid Cache SAN 2.1.0V2.1.0 Storage Deployment and Admin Guide For Linux Systems consists of sections like Table of Contents, to name a few. For easier navigation, use the Table of Contents in the upper left corner.

You can download Dell Fluid Cache SAN 2.1.0V2.1.0 Storage Deployment and Admin Guide For Linux Systems free of charge simply by clicking the “download” button in the upper right corner of any manuals page. This feature allows you to download any manual in a couple of seconds and is generally in PDF format. You can also save a manual for later by adding it to your saved documents in the user profile.

To be able to print Dell Fluid Cache SAN 2.1.0V2.1.0 Storage Deployment and Admin Guide For Linux Systems, simply download the document to your computer. Once downloaded, open the PDF file and print the Dell Fluid Cache SAN 2.1.0V2.1.0 Storage Deployment and Admin Guide For Linux Systems as you would any other document. This can usually be achieved by clicking on “File” and then “Print” from the menu bar.