Contents

Dell ObjectScale V1.0 Storage Administration Guide PDF

1 of 230
1 of 230

Summary of Content for Dell ObjectScale V1.0 Storage Administration Guide PDF

Dell EMC ObjectScale 1.0 Administration Guide

July 2022 Rev. 1.2

Notes, cautions, and warnings

NOTE: A NOTE indicates important information that helps you make better use of your product.

CAUTION: A CAUTION indicates either potential damage to hardware or loss of data and tells you how to avoid

the problem.

WARNING: A WARNING indicates a potential for property damage, personal injury, or death.

2022 Dell Inc. or its subsidiaries. All rights reserved. Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be trademarks of their respective owners.

Revision history........................................................................................................................................................................ 10 Document feedback.......................................................................................................................................................... 10

About using this guide............................................................................................................................................................. 11

Chapter 1: Overview.....................................................................................................................12 About ObjectScale............................................................................................................................................................. 12 Product Description...........................................................................................................................................................12 Basic Terminology.............................................................................................................................................................. 13 ObjectScale User Interfaces............................................................................................................................................14

VMware vSphere with Kubernetes..........................................................................................................................15 ObjectScale Portal UI for Dell EMC ObjectScale for Red Hat OpenShift .....................................................15 Grafana Dashboards.................................................................................................................................................... 15

About installation and user access.................................................................................................................................15 ObjectScale and Kubernetes........................................................................................................................................... 16

ObjectScale Storage Classes.................................................................................................................................... 18 ObjectScale components........................................................................................................................................... 18

Data protection with ObjectScale Erasure Coding schemes...................................................................................19

Chapter 2: Getting Started with ObjectScale............................................................................... 21 Accessing the ObjectScale Instance User Interface................................................................................................. 21

Log in to the ObjectScale Portal UI......................................................................................................................... 21 Log in to VMware vSphere Client UI...................................................................................................................... 23

Navigating within ObjectScale....................................................................................................................................... 23 View Dashboard........................................................................................................................................................... 23 View Accounts............................................................................................................................................................. 24 View Object Stores..................................................................................................................................................... 24 View ObjectScale Systems....................................................................................................................................... 25 View ObjectScale Health........................................................................................................................................... 25 View Settings............................................................................................................................................................... 25

ObjectScale Licensing...................................................................................................................................................... 25

Chapter 3: Working with IAM accounts and users ....................................................................... 28 Accounts..............................................................................................................................................................................28

New Accounts.............................................................................................................................................................. 28 Accounts........................................................................................................................................................................29 Edit Account................................................................................................................................................................. 29 Enable or Disable an Account................................................................................................................................... 30 Delete an account....................................................................................................................................................... 30

Users.................................................................................................................................................................................... 30 Create a IAM user within an account...................................................................................................................... 31 View existing IAM users within an account ......................................................................................................... 32 Edit an IAM user account.......................................................................................................................................... 33 Delete an IAM user account......................................................................................................................................34

Groups..................................................................................................................................................................................35 New Group.................................................................................................................................................................... 35

Contents

Contents 3

Edit Group..................................................................................................................................................................... 35 Delete Groups...............................................................................................................................................................36

Roles.....................................................................................................................................................................................36 New Role....................................................................................................................................................................... 36 Edit Roles.......................................................................................................................................................................38 Delete Roles..................................................................................................................................................................39

Policies................................................................................................................................................................................. 40 Create a new customer-managed policy............................................................................................................... 40 Edit a customer-managed policy.............................................................................................................................. 41 Delete a customer-managed policy.........................................................................................................................43 Actions in IAM Policy.................................................................................................................................................. 44 Principal types in IAM Policies..................................................................................................................................59

Identity Provider................................................................................................................................................................60 New Identity Provider................................................................................................................................................ 60 Edit Identity Providers................................................................................................................................................ 61 Delete Identity Providers............................................................................................................................................61

Root Access Keys.............................................................................................................................................................. 61 Create a Root Access Key......................................................................................................................................... 61 Manage existing Root Access Keys........................................................................................................................ 62

Notification Destinations................................................................................................................................................. 62 Create a notification destination............................................................................................................................. 62 Edit a notification destination...................................................................................................................................63 Delete a notification destination.............................................................................................................................. 64

Metrics.................................................................................................................................................................................64

Chapter 4: Working with object stores........................................................................................ 65 About ObjectScale object stores...................................................................................................................................65

Object store naming conventions............................................................................................................................66 Create an object store..................................................................................................................................................... 67 Associate an IAM Account with an object store....................................................................................................... 69 View object store summary............................................................................................................................................ 70 View S3 and management certificate properties....................................................................................................... 71 Edit an object store........................................................................................................................................................... 71 Set capacity alerts for an object store.........................................................................................................................72 Delete an object store...................................................................................................................................................... 73

Chapter 5: Working with Buckets in ObjectScale ........................................................................ 74 About ObjectScale buckets.............................................................................................................................................74

Bucket and object naming conventions................................................................................................................. 74 Creating and managing buckets using ObjectScale.................................................................................................. 74

Create a bucket............................................................................................................................................................74 View the Bucket Summary........................................................................................................................................ 77 Edit a bucket................................................................................................................................................................. 77 Delete a bucket............................................................................................................................................................ 78

About bucket policies....................................................................................................................................................... 79 Create or edit a bucket policy.................................................................................................................................. 79 Bucket policy support.................................................................................................................................................80 Bucket policy scenarios.............................................................................................................................................. 81 Supported bucket policy operations....................................................................................................................... 82

4 Contents

Supported bucket policy conditions........................................................................................................................83 Setting up bucket event notifications..........................................................................................................................84

About bucket event notifications............................................................................................................................ 87 Configure Webhook Destination for S3 Notifications........................................................................................ 89 Create a bucket notification configuration using the ObjectScale APIs .......................................................90

Chapter 6: Working with ObjectScale Systems............................................................................93 ObjectScale Systems........................................................................................................................................................93 Create a federation of ObjectScale systems..............................................................................................................94 Add additional ObjectScale instances to an existing ObjectScale federation ................................................... 97

Chapter 7: Working with ObjectScale Settings............................................................................ 99 About Settings................................................................................................................................................................... 99 About ObjectScale upgrades..........................................................................................................................................99

Upgrading ObjectScale best practices................................................................................................................. 100 Upgrade to a new version of ObjectScale............................................................................................................101 Upgrade ObjectScale components........................................................................................................................ 102 Upgrading object stores........................................................................................................................................... 103

About SupportAssist....................................................................................................................................................... 104 (Optional) Configure SupportAssist......................................................................................................................104 Edit SupportAssist settings.....................................................................................................................................106 Disable SupportAssist............................................................................................................................................... 106 Remove SupportAssist............................................................................................................................................. 107

Apply the ObjectScale license...................................................................................................................................... 107 SAML Service Provider Metadata............................................................................................................................... 108

Generate SAML Service Provider Metadata.......................................................................................................108 Manage ObjectScale certificates.................................................................................................................................108

Chapter 8: Accessing data with IAM and S3 .............................................................................. 109 ObjectScale Management REST API introduction.................................................................................................. 109

ObjectScale Management REST API summary.................................................................................................. 109 Download and set up CURL..................................................................................................................................... 110 Authenticate with the ObjectScale Management REST API........................................................................... 110

Introduction to Identity and Access Management................................................................................................... 113 Account Management............................................................................................................................................... 114 Access Management.................................................................................................................................................120 Security Token Service.............................................................................................................................................124 IAM SAML support.................................................................................................................................................... 126 IAM Resource ARNs..................................................................................................................................................130

Amazon S3 API support in ObjectScale..................................................................................................................... 130 S3 API support in ObjectScale................................................................................................................................130 ObjectScale S3 error codes..................................................................................................................................... 131 Authenticating with the S3 service....................................................................................................................... 146 Use SDKs to access the S3 service.......................................................................................................................147 Working with S3 workloads in ObjectScale.........................................................................................................150

Chapter 9: Working with ObjectScale Replication...................................................................... 153 Introduction to ObjectScale Replication.................................................................................................................... 153 Bucket Replication Policy.............................................................................................................................................. 153

Contents 5

Manage a Bucket Replication Policy using ObjectScale UI................................................................................... 157 Configure a new bucket replication rule...............................................................................................................157 Edit an existing bucket replication rule................................................................................................................. 161 Delete a bucket replication rule...............................................................................................................................161 Working with bucket replication rules...................................................................................................................162 Change the priority of bucket replication rules.................................................................................................. 163 Configure destination bucket to receive objects............................................................................................... 163

Set up ObjectScale Replication using the ObjectScale API.................................................................................. 165 Create and configure an account and an IAM role............................................................................................ 165 Setup the ObjectScale to ObjectScale Replication........................................................................................... 167

Monitor and manage replication for an object store...............................................................................................168

Chapter 10: Monitoring Events: Audits and Alerts ..................................................................... 170 About ObjectScale instance event and issue monitoring.......................................................................................170

View ObjectScale health issues and events........................................................................................................ 170 View the health of an object store......................................................................................................................... 171

Monitoring Events, Audits, and Alerts........................................................................................................................ 172 CSI-01............................................................................................................................................................................172 CSI-01............................................................................................................................................................................172 CSI-01............................................................................................................................................................................173 CSI-01............................................................................................................................................................................173 CSI-01............................................................................................................................................................................173 CSI-01............................................................................................................................................................................173 CSI-03...........................................................................................................................................................................174 CSI-03...........................................................................................................................................................................174 CSI-04...........................................................................................................................................................................174 CSI-05...........................................................................................................................................................................175 CSI-05...........................................................................................................................................................................175 DECKS-HC-1000........................................................................................................................................................175 DECKS-LIC-1002........................................................................................................................................................175 DECKS-LIC-1005....................................................................................................................................................... 175 DECKS-LIC-1006........................................................................................................................................................176 DECKS-LIC-1008........................................................................................................................................................176 DECKS-LIC-1011......................................................................................................................................................... 176 DECKS-SA-1023.........................................................................................................................................................176 DECKS-SA-1024......................................................................................................................................................... 177 KAHM-HC-1000..........................................................................................................................................................177 OBJSC-LIC-0004....................................................................................................................................................... 177 OBJSC-MGR-3000....................................................................................................................................................178 OBJSC-MGR-HC-1000.............................................................................................................................................178 OBJSC-MON-1111.......................................................................................................................................................178 OBJSC-MON-1112......................................................................................................................................................178 OBJSC-MON-1113......................................................................................................................................................178 OBJSC-MON-3002................................................................................................................................................... 179 OBJSC-MON-3003................................................................................................................................................... 179 OBJSC-MON-4019.................................................................................................................................................... 179 OBJSC-MON-4020................................................................................................................................................... 179 OBJSC-MON-4021.................................................................................................................................................... 180 OBJSC-MON-4022................................................................................................................................................... 180 OBJSC-MON-4025................................................................................................................................................... 180

6 Contents

OBJSC-MON-4028.................................................................................................................................................... 181 OBJSC-SP-0000........................................................................................................................................................ 181 OBJSC-SP-0001......................................................................................................................................................... 181 OBJSC-SP-0002........................................................................................................................................................ 181 OBJSC-SP-0003........................................................................................................................................................182 OBJSOP-1000.............................................................................................................................................................182 OBJSOP-1001............................................................................................................................................................. 182 OBJSOP-1002.............................................................................................................................................................182 OBJSOP-1003.............................................................................................................................................................183 OBJSOP-1004.............................................................................................................................................................183 OBJSOP-1005.............................................................................................................................................................183 OBJSOP-1006.............................................................................................................................................................183 OBJSOP-2001.............................................................................................................................................................184 OBJSOP-2002............................................................................................................................................................ 184 OBJST-1006................................................................................................................................................................ 184 OBJST-1008................................................................................................................................................................ 184 OBJST-12001.............................................................................................................................................................. 185 OBJST-12003............................................................................................................................................................. 185 OBJST-12004............................................................................................................................................................. 185 OBJST-12005............................................................................................................................................................. 185 OBJST-12006............................................................................................................................................................. 186 OBJST-12007..............................................................................................................................................................186 OBJST-13000............................................................................................................................................................. 186 OBJST-13001.............................................................................................................................................................. 186 OBJST-13002..............................................................................................................................................................187 OBJST-13003..............................................................................................................................................................187 OBJST-13004..............................................................................................................................................................187 OBJST-13005..............................................................................................................................................................187 OBJST-13006............................................................................................................................................................. 188 OBJST-13007..............................................................................................................................................................188 OBJST-13008..............................................................................................................................................................188 OBJST-13009............................................................................................................................................................. 188 OBJST-1320................................................................................................................................................................ 188 OBJST-1321................................................................................................................................................................. 189 OBJST-1324................................................................................................................................................................ 189 OBJST-1325................................................................................................................................................................ 189 OBJST-1328................................................................................................................................................................ 189 OBJST-1329................................................................................................................................................................ 190 OBJST-1332................................................................................................................................................................ 190 OBJST-1333................................................................................................................................................................ 190 OBJST-1336................................................................................................................................................................ 190 OBJST-1337................................................................................................................................................................ 190 OBJST-1340.................................................................................................................................................................191 OBJST-1341..................................................................................................................................................................191 OBJST-1344................................................................................................................................................................. 191 OBJST-1345.................................................................................................................................................................191 OBJST-1352.................................................................................................................................................................191 OBJST-1354................................................................................................................................................................ 192 OBJST-1364................................................................................................................................................................ 192 OBJST-1365................................................................................................................................................................ 192

Contents 7

OBJST-1366................................................................................................................................................................ 192 OBJST-1390................................................................................................................................................................ 193 OBJST-1392................................................................................................................................................................ 193 OBJST-1600................................................................................................................................................................193 OBJST-1601.................................................................................................................................................................193 OBJST-1602................................................................................................................................................................ 193 OBJST-1603................................................................................................................................................................ 194 OBJST-1604................................................................................................................................................................ 194 OBJST-1605................................................................................................................................................................194 OBJST-1700................................................................................................................................................................ 194 OBJST-1701.................................................................................................................................................................195 OBJST-MON-4016.................................................................................................................................................... 195 OBJST-MON-4019.................................................................................................................................................... 195 OBJST-MON-4020................................................................................................................................................... 195 OBJSTORE-HC-1000................................................................................................................................................196 SNMPNOTI-1000....................................................................................................................................................... 196 TEST TRAP................................................................................................................................................................. 196

Chapter 11: Viewing ObjectScale and object store metrics..........................................................197 ObjectScale metrics........................................................................................................................................................ 197

Metering details within an ObjectScale instance............................................................................................... 197 ObjectScale metrics in Grafana....................................................................................................................................199

Grafana dashboards overview................................................................................................................................ 199 Navigating Grafana...................................................................................................................................................200 View the Metrics dashboards for the ObjectScale instance...........................................................................201 View the Metrics dashboards for an object store.............................................................................................202

Chapter 12: Troubleshooting and service procedures.................................................................204 About the ObjectScale service pod ...........................................................................................................................204

svc_bucket list.......................................................................................................................................................... 205 svc_bucket info.........................................................................................................................................................206 kpi................................................................................................................................................................................. 206 svc_request................................................................................................................................................................207 svc_log.........................................................................................................................................................................207

Collecting troubleshooting logs................................................................................................................................... 209 Collect vSphere logs................................................................................................................................................ 209

About Service Procedure Operator and ObjectScale service procedures.........................................................210 About ObjectScale capacity expansion procedures.......................................................................................... 210 About maintenance modes...................................................................................................................................... 215 Disk replacement service procedures.................................................................................................................. 220 Node replacement service procedures for vSphere and OpenShift.............................................................222 Troubleshooting Service Procedures................................................................................................................... 224

About creating a new ObjectScale object store using Helm install and a YAML config file......................... 227 Manually create an Object Store using helm install for ObjectScale ...........................................................227

Appendix A: Miscellaneous management tasks for ObjectScale on vSphere............................... 228 Create vSphere namespace and users for ObjectScale........................................................................................ 228

Add vCenter users for ObjectScale......................................................................................................................228 Configure permissions for vSphere end-users...................................................................................................228

8 Contents

Create and configure a new ObjectScale end-user namespace....................................................................229 Create a custom vSAN SNA striped policy for the object store namespace...................................................230

Contents 9

Revision history Table 1. Revision history

Revision Date Revision Number Description of change

July 26, 2022 1.2 Changes include removing specific ObjectScale version and generalizing the document for the 1.0 version and patches.

March 30, 2022 1.1 Changes include improvements for usability of HTML version of the document and corrected order for upgrading ObjectScale components.

February 24, 2022 1.0 Initial release for ObjectScale 1.0.0.

Document feedback If you have any feedback or suggestions regarding this document, objectscale.docfeedback@dell.com.

10 Revision history

About using this guide CAUTION: Adobe Acrobat (Reader, Standard, and Pro) and many other common PDF viewers, including Google

Chrome and Microsoft Edge, add a line break to the end of each line of text within a PDF. This known

limitation results in commands that wrap across multiple lines in the PDF to be copied/pasted with the wrong

format (erroneous line breaks). If these line breaks are in the copied commands, they cause issues during the

installation and administration of ObjectScale.

To ensure that copied commands do not contain unintentional line breaks, it is recommended that you use the

HTML version of this document or paste the copied commands into a text editor and remove the line breaks.

About using this guide 11

Overview This chapter contains:

Topics:

About ObjectScale Product Description Basic Terminology ObjectScale User Interfaces About installation and user access ObjectScale and Kubernetes Data protection with ObjectScale Erasure Coding schemes

About ObjectScale Dell EMC has engineered ObjectScale with a software-defined, containerized architecture to deliver enterprise-class, high performance object storage in a Kubernetes-native package. ObjectScale empowers organizations to move faster and respond effectively to rapidly changing business needs. This next generation of object storage software will be lighter, faster and deployable on existing infrastructure. It supports the storage, manipulation, and analysis of unstructured data on a massive scale.

With rich S3 compatibility and self-service APIs, you can quickly spin up object storage containers to fuel applications ranging from big data and analytics to ephemeral dev or test sandboxes.

Dell EMC ObjectScale allows any organization to deliver scalable cloud services with the reliability and control of a private cloud infrastructure. ObjectScale enables you to easily manage globally distributed storage infrastructure under a single namespace with anywhere access to content.

ObjectScale is built with certain design principles, such as:

Global namespace with eventual consistency Scale-out capabilities Secure multi-tenancy Superior performance for both small and large objects

The platform was built as a distributed system following the microservices principle of cloud applications. ObjectScale has a layered architecture, with every function in the system built as an independent layer, making them horizontally scalable across all nodes and enabling high availability. The S3-compatible ObjectScale software forms the underlying cloud storage service, providing protection, geo-replication, and data access.

Product Description Dell EMC ObjectScale is the next evolution of object storage from Dell EMC and is built to run in Kubernetes and run efficiently on shared infrastructure and in multi-tenant environments.

Dell EMC ObjectScale gives organizations the power to put data closer to the applications they support, reducing latency and improving the user experience. In addition, object storage from disparate platforms can cross-replicate for greater access, reliability, and redundancy.

ObjectScale is an enterprise-grade object storage with these features and functionality:

Simple, S3-compatible enterprise-grade object storage Kubernetes-based, customer-deployable on both VMware vSphere with Tanzu and Red Hat OpenShift Scaled-out, software-defined architecture

Also, other major changes that are introduced with ObjectScale include:

1

12 Overview

Improved data protection with new erasure coding schemas, such as 3+3 in three node deployments

New replication model with eventual consistency for greater availability during hardware failure Integrated Pravega to manage bucket or object events enabling bucket notifications, ObjectScale replication, and metering A complete multi-tenant IAM service with IAM accounts and other IAM entities, like Users, Groups, Roles, Policies, and

Service Providers

Building ObjectScale for Kubernetes allowed Dell EMC to deliver a simplified product where Kubernetes handles the OS- and hardware-level layers leaving ObjectScale to handle the storage and storage management.

With this underlying Kubernetes architecture, ObjectScale gives you segmented control of the storage, compute, and network services and allows for dynamic provisioning of resources. You can control when new services are started on an as needed basis. These new resources are tied to the underlying resources upon creation.

Here are some of the benefits Kubernetes provides for ObjectScale:

Predictable application deployment using a declarative method Dynamic scaling of application resources Deployment using only required resources Highly portable across deployment models Self-healing: Autoplacement, auto restart, and autoreplication

In Kubernetes, each resource can be affinitized to run on one host. Affinization of resources to hosts allows ObjectScale to behave as its own fault domain.

Kubernetes is responsible for operating system and hardware interaction. The customer is responsible for TOR switching and network infrastructure. Networking inside Kubernetes is provided by a Container Networking Interface (CNI) which for VMware is implemented using NSX-T. In OpenShift network services are provided by MetalLB or other customer-provided Kubernetes compatible load-balancing application.

Flexible deployment environments are central to ObjectScale as it provides increased flexibility for customer-built and maintained object storage platform. ObjectScale performance and maintenance are aided by the collocation of the compute and storage infrastructure.

Basic Terminology The following terms are basic to understand ObjectScale.

Account A logical construct that corresponds to a customer business unit, tenant, project, and so on, which are relevant to the account admin role and end users that belong to an account.

Admin Admin of an ObjectScale or a federation of ObjectScale instances.

Buckets Buckets are object containers that are used to control access to objects.

Chunk A Chunk is the basic unit in ObjectScale for data storage and protection. A chunk is 128MiB of logical storage that is erasure-coded and written to multiple disks across multiple nodes in the instance.

Custom Resource Definition

Custom Resource Definitions are extensions to Kubernetes API resources. ObjectScale adds CRDs that create custom resources with the specified name and schema.

DECKS Dell EMC Common Kubernetes Services created by Dell EMC.

Federation A federation joins multiple ObjectScale instances together. Global information like endpoints or global accounts are replicated throughout an ObjectScale federation.

Horizontal Expansion

Object stores can be expanded through horizontal expansion by adding more Storage Servers to the object store.

IAM Role An IAM Role (role) is an IAM identity that you can create in your account that has specific permissions. An IAM Role is similar to an IAM end user, in that it is an ObjectScale identity with permission policies that determine what the identity can and cannot do in ObjectScale. However, instead of being uniquely associated with one person, a role is intended to be assumable by anyone who needs it.

KAHM Kubernetes Application Health Management created by Dell EMC.

Kubernetes Kubernetes (K8S) is an open-source container-orchestration system for automating application deployment, scaling, and management.

Overview 13

Large Size Object A large object is one where most of the I/O time is spent accessing the data of the object, and thus is bounded by the performance of the data path and disks. Large objects are 10MiB or more. The inflection point between the two is around 1MiB depending on system configuration.

Namespace In Kubernetes, namespaces act as a mechanism for isolating groups of resources within a single cluster.

Normal Size Object

Object with size close to many write blocks. Read data for the objects generate large sequential read I/O on disk.

Object Attribute An object attribute is an aspect of an object version that can be updated and replicated separately, such as an object tag, ACL, or lock.

Object Data Data of an object version.

Object Data Index

The data locations of an object version on chunks.

ObjectScale Instance

ObjectScale is deployed in a Kubernetes cluster. The deployment is termed as an ObjectScale Instance (OSI). ObjectScale, or the ObjectScale instance, is a software bundle of management services that contains everything that is must deploy and consume Dell EMC object storage. The ObjectScale instance is deployed once per Kubernetes cluster and provides management and shared object store services. Including: IAM Federation service Serviceability features such as SupportAssist.

Object Stores A unique and independent storage system with an individualized life cycle. One or more object stores are deployed by each ObjectScale instance. Object stores are created, updated, and deleted independently from all other object stores managed by the shared ObjectScale instance. Kubernetes cluster resources such as storage, CPU, and RAM are defined for each object store based on workload demand inputs that are specified at object store creation. Resources that are reserved for an object store at creation may be adjusted at any time.

Object Metadata The system or user metadata of an object. Object metadata is a part of the object version, and it cannot be updated separately. As a result object metadata is not a part of ObjectScale replication attributes. Object metadata is replicated whenever the object data is replicated.

Object Version All data or metadata or attribute belongs to a specific version of an object.

Resource Names Resource Names (RNs) are names that uniquely identify resources. Resource Names (RNs) are required when user must specify a resource unambiguously in an ObjectScale.

Small Size Object Small objects are about 100 KB or less. A small object is one where most of the I/O time is spent accessing metadata and thus, is bounded by the performance of the metadata services.

Storage Class Storage Class determines which driver is used to create a persistent volume. At a per ObjectScale level, the admin can map storage classes to storage tiers.

Storage Servers (SS)

Storage Servers (SS) in ObjectScale interact with storage media. In ObjectScale, each physical server is a Kubernetes node, and each SS pod instance is an ObjectScale node.

SupportAssist SupportAssist provides a network based connection to Dell Support. SupportAssist enables Dell Support to receive telemetry and issues, events, and alerts from your ObjectScale instance, and to perform remote troubleshooting, resulting in a fast and efficient time to resolution.

Tenant A tenant is a logical construct resulting from the binding of an IAM account to an object store. When an IAM account is added to an object store, that account becomes a tenant within that object store.

Vertical Expansion

ObjectScale can be expanded through vertical expansion by increasing the number of volumes per Storage Server replica in the object store.

ObjectScale User Interfaces Dell EMC ObjectScale provides the following interfaces. VMware vSphere with Kubernetes ObjectScale Portal UI for Dell EMC ObjectScale for Red Hat OpenShift Grafana Dashboards

14 Overview

VMware vSphere with Kubernetes

VMware vSphere is virtualization platform of VMware, which transforms data centers into aggregated computing infrastructures that include CPU, storage, and networking resources. vSphere manages these infrastructures as a unified operating environment, and provides you with the tools to administer the data centers that participate in that environment.

You can use vSphere with Kubernetes to transform vSphere to a platform for running Kubernetes workloads natively on the hypervisor layer. When enabled on a vSphere cluster, vSphere with Kubernetes provides the capability to run Kubernetes workloads directly on ESXi hosts and to create upstream Kubernetes clusters within dedicated resource pools.

VMware has also refactored vSphere beginning in version 7.0 to leverage Kubernetes in their control plane. The integration of Kubernetes within vSphere is a result of a VMware internal project that introduced Workload Management (cluster) and Workload Control Plane (WCP). These terms generally are used to describe Kubernetes functionality within the vSphere product.

For details, see Log in to VMware vSphere Client UI.

ObjectScale Portal UI for Dell EMC ObjectScale for Red Hat OpenShift

The ObjectScale Portal UI is used to manage deployments of Dell EMC ObjectScale for Red Hat OpenShift. The Red Hat OpenShift Container Platform is for developing and running containerized applications and is designed to easily allow applications to expand as needed.

Per Red Hat:

"OpenShift Container Platform provides enterprise-ready enhancements to Kubernetes, including the following enhancements: Hybrid cloud deployments: You can deploy OpenShift Container Platform clusters to various public cloud platforms or in

your data center. Integrated Red Hat technology: Major components in OpenShift Container Platform come from Red Hat Enterprise Linux

and related Red Hat technologies. OpenShift Container Platform benefits from the intense testing and certification initiatives for enterprise quality software by Red Hat.

Open-source development model: Development is completed in the open, and the source code is available from public software repositories. This open collaboration fosters rapid innovation and development.

"

The Dell EMC ObjectScale Portal UI for this OpenShift cluster environment allows you to easily manage the ObjectScale object storage within your OpenShift environment.

Grafana Dashboards

ObjectScale includes the collection, storage, and visualization of detailed metrics in Grafana dashboards. Administrators can use these dashboards to drill into problems or identify developing problems with ObjectScale or problems with underlying storage resources.

Similar metrics are also available at the ObjectScale-level, from the Dashboard page (must have admin permissions to see them).

Grafana is an open-source metrics visualization tool. The ObjectScale installation deploys Grafana.

See Grafana for basic details of navigation in Grafana dashboards.

About installation and user access

Your cluster administrator installs and, by default, can manage all aspects of the ObjectScale instance. The cluster administrator user can set up end-user namespaces and/or users with specialized permissions for access to object stores deployed within an end-user namespace.

The administrator users have access to all of the ObjectScale instance, Identity and Access Management, object stores, and Grafana dashboards, by default. Depending on the user permissions within the cluster, object stores are created and managed

Overview 15

by end-users and administrators. User roles can be configured to allow end-users and administrators to monitor the resource usage associated with object stores and the ObjectScale instance using the ObjectScale UI.

User access roles

ObjectScale provides a user interface (UI) for all users interacting with object stores and the ObjectScale instance.

Access roles control the views and actions available to each user. User accounts must be set up in the cluster for users to access the ObjectScale UI.

Table 2. User access roles

Role Persona Activities

ObjectScale for vSphere and ObjectScale for OpenShift Administrator

ObjectScale Instance Administrator

Create namespaces and users Create object stores and buckets Create and maintain IAM entities Manage the federation of ObjectScale instances and

replication Monitor ObjectScale performance, storage, and resource

allocation Monitor ObjectScale instance and object store Grafana

dashboards Perform ObjectScale upgrade and maintenance activities

ObjectScale Administrator

ObjectScale User with edit access to ObjectScale and other namespaces

Edit access to ObjectScale default namespace (or ObjectScale manager namespace) and other namespaces which they can access

Create object stores and buckets Create and maintain IAM entities Manage the federation of ObjectScale instances and

replication Monitor ObjectScale performance, storage, and resource

allocation Monitor ObjectScale instance and object store Grafana

dashboards

End-User with Edit role Object store user Create object stores and buckets in the namespace which they can access

Monitor performance, storage, and resource allocation in the namespace where the permissions were given to the user

Monitor object store Grafana dashboards

End-User with view or read-only

Cluster or ObjectScale monitor View performance, storage, and resource allocation in the namespace where the permissions were given to the user

Monitor object store Grafana dashboards

ObjectScale and Kubernetes ObjectScale is a software bundle of management services that contain everything that is must deploy and consume Dell EMC object storage. ObjectScale is deployed in a Kubernetes cluster allowing Kubernetes to handle the necessary orchestration. One ObjectScale instance with one or more object stores is deployed per Kubernetes cluster.

Kubernetes is an open-source container-orchestration system for automating application deployment, scaling, and management. A Kubernetes cluster consists of physical or virtual nodes. Each Kubernetes node runs a process that is named kubelet. ObjectScale is built on Kubernetes clusters using physical server infrastructure.

In ObjectScale, Kubernetes provides the connective glue between physical infrastructure, such as disk and network, and the application services running in containers. ObjectScale leverages on the efficient resource management capabilities of Kubernetes and relies on it to handle operating system and hardware interaction.

16 Overview

ObjectScale Kubernetes components

ObjectScale includes these software components:

1. ObjectScale ManagerInstalls and manages the custom ObjectScale resources. 2. Dell EMC Common Kubernetes Services (DECKS)A suite of tools such as Dell SupportAssist Embedded Support Enabler

(ESE) for log collection and license resource consumption telemetry. 3. Kubernetes Application Health Management (KAHM)Event persistence management, notifications, and complex event

routing rules 4. User Interfaces:

ObjectScale UI Plug-in for the VMware vSphere UI A stand-alone ObjectScale UI (OpenShift deployments) Grafana, with preconfigured dashboards for monitoring the ObjectScale instance Kubectl plug-in at CLI Helm binary at CLI (OpenShift deployments) ObjjectScale Management API

Operators and ObjectScale

Kubernetes has a concept that is called an Operator. An Operator is an application-specific controller and contains all the operational considerations of an application. Operator resources are defined in YAML files as Kubernetes Custom Resource Definitions (CRD). Custom resources define actions available to users of the Operator. Kubernetes manages custom resources like it manages its own integrated resources.

The ObjectScale operator is a custom resource defined that can create object stores. The ObjectScale operator connects object stores to the management services, and orchestrates operations, such as upgrades and deletions.

The ObjectScale Service Procedure operator is a custom resource defined that helps orchestrate service-oriented tasks such as temporary maintenance mode, disk replacements, object store expansions, and upgrades.

The ZooKeeper operator is a custom resource that is defined to manage the Pravega ZooKeeper cluster and all the ZooKeeper clusters for ObjectScale.

The Atlas operator is a custom resource defined:

To provide Atlas services to implement a new key-value store To provide better stability, predictability, and efficiency. In terms of per-operation overhead (key-value operations), system

operation overhead (node replacement), and overall CPU and memory use.

Other Kubernetes resources to know

Here is a list of additional common Kubernetes resources for administrators of ObjectScale:

Annotations are key-value maps used to attach arbitrary nonidentifying metadata to objects such as Pods. Annotations are used by tools and libraries.

Labels are key-value pairs that are attached to objects. Labels are used to organize and to select subsets of objects. Pods are a unit of application running in Kubernetes. Each pod is consisted of one or more containers. A set of pods makes

up a Kubernetes application. ObjectScale deploys 17 types of pods for each object store. A Deployment provides declarative updates for Pods and ReplicaSets. A deployment describes a wanted state. The

deployment controller tracks and maintains actual state to the wanted state. A ReplicaSet is a deployment model available in Kubernetes. A ReplicaSet is a set of one or more of a single type pod.

ReplicaSets are used to guarantee availability of the service they provide. An example of a ReplicaSet used in ObjectScale is GraphQL. The number of replicas in a set may be adjusted on the fly.

A StatefulSet is a deployment model in Kubernetes. StatefulSets are used for deploying state-full applications. StatefulSets manage the deployment and scaling of a set of Pods and provide guarantees about the ordering and uniqueness of these Pods. StatefulSets maintain a sticky identity to a Kubernetes node for each pod in the set.

A Service is an abstract way to expose an application running on a set of Pods as a network service. Networking services are provided for Kubernetes environments that allow for ingress, egress, and load balancing of traffic in and out of the Kubernetes environment. Client connectivity to ObjectScale is provided by Services.

A PersistentVolume (PV) is storage that is provisioned on available storage.

Overview 17

A PersistentVolumeClaim (PVC) is a request for PV resources. PVCs request and consume specific size and access modes. A PVC, or claim for short, is bound to a persistent volume. Persistent Volumes and associated provisioned virtual disks are deleted at PVC deletion. Pod creation and destruction have no effect on PVC or PV.

ObjectScale Storage Classes

Review for details on the storage classes (SC) for each ObjectScale deployment model.

OpenShift and Dell EMC bare-metal CSI Driver Storage Classes

Name Reclaim Policy Volume Binding Mode

Allow Volume Expansion

Disk Micro Partitioning

Highly Available

Media Types

csi-baremetal-sc (default)

Delete WaitForFirstCon sumer

No No No Any

csi-baremetal- sc-hdd

Delete WaitForFirstCon sumer

No No No HDD

csi-baremetal- sc-hddlvg

Delete WaitForFirstCon sumer

Yes Yes No HDD

csi-baremetal- sc-nvme

Delete WaitForFirstCon sumer

No No No NVMe

csi-baremetal- sc-ssd

Delete WaitForFirstCon sumer

No No No SSD

csi-baremetal- sc-ssdlvg

Delete WaitForFirstCon sumer

Yes Yes No SSD

csi-baremetal- sc-syslvg

Delete WaitForFirstCon sumer

Yes Yes No Any

vSphere vSAN CSI Storage Classes

Name Reclaim Policy Volume Binding Mode

Allow Volume Expansion

Disk Micro Partitioning

Highly Available

Media Types

highly-available Delete Immediate No Yes Yes Any

vsan-direct- thick

Delete WaitForFirstCon sumer

No Yes No Any

vsan-sna-thick Delete WaitForFirstCon sumer

No Yes No Any

ObjectScale components

A deployment contains these components to support ObjectScale features and functionality.

Review the table for information about the various ObjectScale and object store components that are deployed with an ObjectScale instances and its size and expected storage classes (SC).

Name Level Highly Availabl e

Minimu m replicas count

Volume Size(large profile)

SSD vSphere SC OpenShift SC

rsyslog ObjectS cale

No Number of nodes

Default 200Gi No vsan-sna-thick csi-baremetal-sc- hddlvg

iam-atlas ObjectS cale

Yes 3 Default 10Gi Yes highly-available csi-baremetal-sc- ssdlvg

18 Overview

Name Level Highly Availabl e

Minimu m replicas count

Volume Size(large profile)

SSD vSphere SC OpenShift SC

dcm-atlas ObjectS cale

Yes 3 Default 1Gi Yes highly-available csi-baremetal-sc- ssdlvg

federation- atlas

ObjectS cale

Yes 3 Default 10Gi Yes highly-available csi-baremetal-sc- ssdlvg

influxdb ObjectS cale

Yes 3 Default 20Gi Yes highly-available csi-baremetal-sc- ssdlvg

db-kahm ObjectS cale

Yes 3 Default 8G i No highly-available csi-baremetal-sc- ssdlvg

decks- support-store

ObjectS cale

No 1 Default 200Gi No highly-available csi-baremetal-sc- ssdlvg

supportassist var/config

ObjectS cale

No 1 Default 2Gi No highly-available csi-baremetal-sc- ssdlvg

supportassist suport-store

ObjectS cale

No 1 Default 50Gi No highly-available csi-baremetal-sc- ssdlvg

ss object store

No 3 Multiple Varies No vsan-sna-thick csi-baremetal-sc-hdd

atlas object store

Yes 3 Default 32Gi Yes highly-available csi-baremetal-sc- ssdlvg

zookeeper object store

Yes SS<3: 1

3 <5: 3

SS>4: 5

Default 2Gi No highly-available csi-baremetal-sc- ssdlvg

influxdb object store

Yes 3 Default 20Gi Yes highly-available csi-baremetal-sc- ssdlvg

bookie object store

Yes SS<3: 1

SS=3: 3

SS>4: max(4, 0.5 * SS)

index 3Gi Yes highly-available csi-baremetal-sc- ssdlvg/hddlvg

journal 50Gi Yes highly-available csi-baremetal-sc- ssdlvg/hddlvg

ledger 300Gi Yes highly-available csi-baremetal-sc- ssdlvg/hddlvg

Data protection with ObjectScale Erasure Coding schemes ObjectScale uses various Erasure Coding schemes for data protection.

During the object store creation process, the available EC schemes that are presented within the New Object Store wizard is based on the number of Kubernetes nodes, either physical servers (vSphere) or worker nodes (OpenShift), in the cluster. ObjectScale uses the Kubernetes anti-affinity rules to ensure that the SS replicas are properly placed across the nodes in the cluster. The New Object Store sizer ensures that the number of SS replicas is not below the minimum for the selected EC scheme.

A minimum of three physical servers are required to use ObjectScale data protection.

Overview 19

Supported ObjectScale Erasure Coding Schemes

Table 3. ObjectScale on vSphere

Deployed By 12:4 3:3 10:2

Deployed By Helm Supported Supported Supported

Deployed By UI Supported Supported Not Supported

Table 4. ObjectScale on OpenShift

Deployed By 12:4 3:3 10:2

Deployed By Helm Supported Supported Supported

Deployed By UI Supported Supported Not Supported

Minimum Node or Partition requirement for various EC schemes:

12+4: Min: Five nodes, each node at least five partitions. Storage protection overhead: 1.33x 3+3: Min: Three nodes, each node at least four partitions. Storage protection overhead: 2.0x

NOTE: Dell does not recommend using three node clusters in an enterprise or production environment. Currently

upgrades on an ObjectScale system with three nodes are disruptive and requires a manual workaround.

10+2: Min: Seven nodes, each node at least three partitions

NOTE: Can only be deployed and maintained using helm.

An object store with a 3+3 EC scheme uses three code and three data blocks for data protection. This means every user data that are written in to the object store is split into three segments and each segment is written to unique host or SS pod. Each of the three data segments is protected by a copy that is written to another node.

20 Overview

Getting Started with ObjectScale Use these sections to begin using ObjectScale following installation.

Topics:

Accessing the ObjectScale Instance User Interface Navigating within ObjectScale ObjectScale Licensing

Accessing the ObjectScale Instance User Interface In this ObjectScale , ObjectScale can be deployed on Red Hat OpenShift and VMware vSphere with Tanzu clusters.

The type of cluster environment your ObjectScale instance is deployed within determines how you access the ObjectScale User Interface.

For ObjectScale on OpenShift, follow Log in to the ObjectScale Portal UI. For ObjectScale on vSphere, follow Log in to VMware vSphere Client UI.

Log in to the ObjectScale Portal UI

To access the ObjectScale plug-in on an OpenShift cluster, follow these steps to connect to the ObjectScale Portal UI with a supported internet browser.

Prerequisites

If you have not already done so, obtain the network address (EXTERNAL-IP) of the ObjectScale UI:

kubectl -n get svc objectscale-portal-external

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE objectscale-portal-external LoadBalancer 10.55.66.100 10.x.y.z 4443:30436/TCP 9m2s

Steps

1. Open a supported web browser and connect to the ObjectScale Portal UI URL using the External IP address and port for the objectscale-portal service:

https:// :4443

The ObjectScale Portal UI login page appears.

2. Use your Username and Password credentials to log in to the ObjectScale Portal UI.

For OpenShift, obtain your credentials using the OpenShift identity provider user that is configured for accessing the ObjectScale instance and, or object store.

You can log in as kubeadmin or an OpenShift Identity Provider User cluster-admin and can have access to the entire ObjectScale instance and you can create object stores in any namespace you have configured.

You can log in as an OpenShift Identity Provider User end user and can access only the namespaces to which you have permissions.

If you log in as an end user who has edit permissions to a specific namespace, you can manage the object store in that namespace. All other ObjectScale features will not be available.

If you log in as an end user who has view permissions to a specific namespace, you can view the Object Store in that namespace. All other ObjectScale features will not be available.

2

Getting Started with ObjectScale 21

ObjectScale Portal UI

When launched, the ObjectScale Portal UI Dashboard page appears.

Figure 1. ObjectScale Portal UI Dashboard

From the dashboard, you can easily go to detailed views into the created object store by clicking Object Stores and then clicking the name of the object store.

This Dashboard page displays an at-a-glance view of the details about the health and usage of the selected object store.

22 Getting Started with ObjectScale

Figure 2. Dashboard

Scroll down the object store Dashboard page to locate details about the Health, Capacity Utilization, System Data, and Data Management.

Log in to VMware vSphere Client UI

This task describes how to log in to the VMware vSphere Client UI and to go to the ObjectScale Dashboard page.

Steps

1. Log in to VMware vSphere Client UI.

Use your Username and Password to log in to VMware vSphere Client UI.

2. Go to the Inventory, and then select the cluster that is configured for Workload Management.

3. Click the Configure tab and then scroll and select Dashboard listed under the ObjectScale section.

Navigating within ObjectScale Use the following sections to navigate within ObjectScale.

View Dashboard

The Dashboard tab provides an overview of ObjectScale. In ObjectScale Portal UI, the Dashboard tab is the default tab that is open. In VMware vSphere Client UI, to view Dashboard, go to Configure > ObjectScale > Dashboard.

Select the namespace from the namespace drop-down on the upper right corner of the ObjectScale UI.

Object Store Performance Name State Latency Compression Ratio

ObjectScale Summary Health - Monitor the ObjectScale system alerts. Click links in the row to see related alerts. System Data - The capacity that is used by the ObjectScale processes that track and describe the data in the system.

Hover over the category names to know more.

Getting Started with ObjectScale 23

Capacity Utilization - Monitor all capacities at the ObjectScale level. Hover over the category names to know more. Data Management - Monitor the capacity reclaimed, user data reclaimed, and system metadata reclaimed in

ObjectScale level. Hover over the category names to know more.

Table 5. Dashboard field details

Field Description

Name Lists the object stores present in the namespace that is selected in the upper right corner of the object store performance section.

State Lists the states of object stores

Latency Latency consists of: Read First Byte(p50) Write Last Byte(p50) Read First Byte(p99) Write Last Byte(p99)

Compression Ratio Lists the compression ratios of object stores

Health The Health section shows information about number of unacknowledged issues with severity: Critical Error Warning

System Data The System Data section shows information that is related to data such as: Data Protection Metadata Metadata Protection Data pending for EC Rate of EC per Second

Capacity Utilization The Capacity Utilization section shows information that is related to capacity such as: Physical Used Available Reserved Total % Full Days until Full (Est)

Data Management The Data Management section shows information that is related to data such as: Data Being Reclaimed Unreclaimable Metadata Unreclaimable User Data Reclaimable Metadata Reclaimable User Data Capacity Reclaimed

View Accounts

The Accounts tab provides an overview of the available accounts in Object Stores, and also allows you to create, edit, enable, and disable accounts.

For details, see Accounts.

View Object Stores

The Object Stores tab provides an overview of the available object stores in ObjectScale, and also allows you to create, edit, and delete object stores.

For details, see Working with object stores.

24 Getting Started with ObjectScale

View ObjectScale Systems

The ObjectScale Systems tab provides an overview of the available ObjectScale instances in the federation, and also allows you to create, and join federations.

For details, see ObjectScale Systems.

View ObjectScale Health

ObjectScale and object store health alerting are in different sections of the ObjectScale instance.

For details, see Monitoring Events: Audits and Alerts .

ObjectScale Health

The Health tab displays the alerts and issues for the ObjectScale.

ObjectScale > Health

NOTE: In the vSphere Client, this page is at Workload Cluster > Monitor > ObjectScale > Health.

Object store Health

The object store Health tab displays the alerts and issues for the selected object store. Only object store level health messages are tracked and displayed here. This page does not contain health messages about the overall ObjectScale instance.

Object Store > > Health

NOTE: In the vSphere Client, this page is at Workload Cluster > Configure > ObjectScale > Object Store > Store Name> > Health.

View Settings

The Settings tab consists of the various settings options that you can view and configure in ObjectScale.

The Settings tab includes:

Upgrades SupportAssist Licensing SAML Service Provider Metadata ObjectScale Certificate

For details, see Working with ObjectScale Settings.

ObjectScale Licensing Dell EMC ObjectScale provides various deployment models with different licensing options.

The ObjectScale requires a valid license in order to create an object store or to configure SupportAssist. The types of license available for ObjectScale are:

Permanent Subscription Evaluation Community Edition

Getting Started with ObjectScale 25

Permanent license

ObjectScale supports a permanent license. Customers using a permanent license have full access to all ObjectScale features and capabilities and the license does not expire. After purchasing the license, you must activate the license at https:// licensing.emc.com/.

Customers with valid site ids can configure SupportAssist and rely on all its features.

You can apply an new Permanent license, to expand capacity as necessary, retaining the object stores, buckets, and other settings and configurations made to ObjectScale while the license was applied.

Subscription license

ObjectScale supports a subscription license. Customers using a subscription license have access to all ObjectScale features and capabilities, up to the subscribed capacity, until the subscription is no longer active. After purchasing the license, you must activate the license at https://licensing.emc.com/.

Customers with valid site ids can configure SupportAssist and rely on all its features.

You can apply an expanded or updated Subscription or Permanent license, retaining the object stores, buckets, and other settings and configurations made to ObjectScale while the license was applied.

Evaluation license

ObjectScale supports an evaluation license. The evaluation license acts as a short-term license for trials or evaluation of ObjectScale. The evaluation license does not require activation before using in ObjectScale.

Evaluation licenses can carry restrictions for how ObjectScale and object stores can be configured, and the period that the license is valid. These attributes are described within the license file and are enforced by ObjectScale, until the license is no longer valid.

If you must extend an evaluation license for a longer period of time or change the licensed capacity, request a new evaluation license and apply to the ObjectScale instance.

You can apply a Subscription, or Permanent license, retaining the object stores, buckets, and other settings and configurations made to ObjectScale while the license was applied.

Community Edition capacity-limited license

Dell EMC provides a Community Edition, capacity-limited license on the ObjectScale product page in Dell Support (https:// www.dell.com/support/home/en-us/product-support/product/objectscale/drivers). This Community Edition license does not expire, but is limited to a maximum overall capacity of 30TiB. The Community Edition license does not require activation before using in ObjectScale.

ObjectScale instances using the Community Edition license must meet the minimum hardware and software requirements that are found in the ObjectScale installation guides.

Customers with valid site ids can configure SupportAssist and allow it to send telemetry dial homes back to Dell Support. SupportAssist on ObjectScale instances using the Community Edition license does not allow you to send issues or allow remote dial-in via Dell Support.

You can apply an Evaluation, Subscription, or Permanent license, retaining the object stores, buckets, and other settings and configurations made to ObjectScale when the license was applied.

More details about activating purchased ObjectScale licenses

After purchasing an ObjectScale license, Dell EMC sends License Activation Code (LAC) letter to the customer-provided email address associated with the Dell Support account. This email contains the necessary information and steps that you must follow to activate the ObjectScale license.

Go to https://licensing.emc.com/ and follow the online process to generate license files or keys from their entitlements. License activation occurs after Customers or Partners receive a License Authorization Code (LAC) letter and the LAC number that is listed on their LAC letter is redeemed.

26 Getting Started with ObjectScale

When Dell EMC issues new license entitlements to a customer based on a purchase, evaluation, or other event, the entitlements are associated to a unique License Authorization Code (LAC).

A LAC can have one or more entitlements that are associated to it. A LAC is the primary identifier of the entitlements, which the customer or partner uses to locate and activate licenses.

When a LAC is generated, an email is sent to the customer, but it is not always the case for evaluation licenses or for purchases of certain products.

When your software order is fulfilled, you receive an email or letter that includes the LAC for your order and instructions for activating entitlements online.

If you have any questions about your Dell EMC order, contact your Dell EMC Sales Account Representative or your Authorized Reseller.

If you have any questions about Dell EMC software licensing, contact the Licensing Support team.

Once the license is activated, Dell EMC Licensing sends the Dell EMC software license activation notification email to notify that software licenses that are associated with your Dell EMC account are activated.

Review the details within the email and contact Dell support if you think this license activation is erroneous or unintended.

The activated ObjectScale license .xml file is attached to this email notification. Use this license .xml file within the ObjectScale UI to activate the product. For more information, see Apply the ObjectScale license.

Getting Started with ObjectScale 27

Working with IAM accounts and users This chapter contains:

Topics:

Accounts Users Groups Roles Policies Identity Provider Root Access Keys Notification Destinations Metrics

Accounts ObjectScale users with the Admin role can create accounts in an ObjectScale instance. Once created, the Admin must assign the Accounts to object stores for the account to create buckets in the object store.

This means that for a dedicated Object store there is only one tenant or account while a shared object store can have more than one tenant.

Once a bucket is created by an account, that account owns the bucket and can assign permissions to other accounts for cross-account access.

NOTE: An account is not required to be assigned to the object store for it to create objects in a bucket in the object

store.

An account can be assigned to multiple object stores.

On Account page in a properly credentialed user can see a list of accounts with following details for each account:

Alias Account ID Enabled Created On Description Groups Users Roles

Also, the user can perform the following Accounts actions using the ObjectScale Portal UI:

Create account. Edit existing account. Enable account. Disable account.

New Accounts

This task describes how to create an Account in ObjectScale.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts.

3

28 Working with IAM accounts and users

For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts.

For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Select NEW ACCOUNT. New Account window opens.

4. Fill all the required fields in the New Account page.

Field Description

Alias An informal name for the new account.

Description Enter details about the new account.

Encryption Encryption is disabled by default.

Click to enable or disable encryption.

5. Click SAVE. The new Account is saved with added authorization for buckets.

Accounts

This task describes how to view Accounts in ObjectScale.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. To view details of an Account, click the name of an account that is listed in the accounts table. The account details page consists of: The Summary tab displays by default and shows details about the account and the account data. The Buckets tab displays the list of buckets for the first object store. See Buckets, for details. The Users tab displays the list of all IAM Users in the Account. See Users, for details. The Groups tab displays the list of all IAM Groups in the Account. See Groups, for details. The Roles tab displays the list of all IAM Roles in the Account. See Roles, for details. The Policies tab displays the list of all IAM Policies in the Account. See Policies, for details. The Identity Provider tab displays the list of all IAM Identity Providers in the Account. See Identity Provider, for details. The Root Access Key tab displays the list of all Root Access Key details for the Account. See Root Access Key, for

details. The Notification Destinations tab displays the list of all the Notification Destinations in the Account. See Notification

Destinations, for details. The Metrics tab displays Account metrics on a Grafana UI. See Metrics, for details.

Edit Account

This task describes how to edit an Account in ObjectScale.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab.

Working with IAM accounts and users 29

A user can see a list of Accounts that the user is authorized to view.

3. Select an account and click EDIT. A new window opens.

4. Edit the fields in the new window.

NOTE: All fields are not editable.

Field Edit option

Alias Alias of the account is not editable.

Description Description about the new account is editable.

Encryption Encryption is not editable.

Status Status is not editable.

5. Click SAVE. The Account is saved with updated fields.

Enable or Disable an Account

This task describes how to enable or disable an Account in ObjectScale.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Select an account and click ENABLE or DISABLE.

Delete an account

This task describes how to delete an Account in ObjectScale. You cannot delete an account using the UI. Accounts can only be deleted using the IAM API.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Dashboard: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Dashboard. For ObjectScale Portal UI: Click the Dashboard tab. The ObjectScale Dashboard page is displayed showing an overview of the ObjectScale instance.

3. Follow the steps in Delete a bucket to delete a bucket and its objects.

4. Follow the steps in Delete an IAM user account to delete all the IAM user accounts from the IAM account.

5. Finally, use the IAM API to remove the IAM account.

The Dell EMC ObjectScale RestAPI zip file with the supported APIs is available at https://www.dell.com/support/home/ product-support/product/objectscale/drivers.

Users In ObjectScale, an IAM User is a person or application in the account.

Use the following tasks to manage ObjectScale IAM users.

30 Working with IAM accounts and users

Create a IAM user within an account

This task describes how to add an IAM user in ObjectScale.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Select an account from the account list and then select the Users tab. The Users list appears.

4. Click New User. The New User window opens.

5. Fill all the required fields in the New User window.

Field Descriptions

Name a. Enter name of the user. b. To go to Permissions tab, click Next.

Permissions a. You can add permissions to the new user in one of the following ways: Permission boundary and copy permission from an existing user to the current account. Add existing group of current account and permission boundary. Add existing policies of current account and permission boundary.

b. To go to Tags tab, click Next.

Tags (Optional)

You can add one or more tags to a User. a. Enter the details for Key and Value for a tag. b. To go to Review tab, click Next.

Review a. Review details of the user. b. Click CREATE USER.

Secret Keys The Secret Keys tab consists of: A list of users that are created along with permission, Access Key ID, and Access Secret keys. To download the user table in CSV format, click DOWNLOAD.CSV button.

NOTE: This is the only time that you will be able to download this .csv for this user.

Working with IAM accounts and users 31

Figure 3. New User - Secret Keys tab

6. Click Complete. A new IAM user added in ObjectScale.

View existing IAM users within an account

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Select an account from the account list and then select the Users tab.

The list of Users within that account appears.

32 Working with IAM accounts and users

Figure 4. Accounts > Users

Edit an IAM user account

This task describes how to edit details of an IAM user in ObjectScale.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Select an account from the account list and select Users. Select a user from the user list. The Summary tab opens by default, and is not editable. The other tabs are: Groups Permissions Tags Secret Key

4. Edit the fields.

NOTE: All fields are not editable.

Table 6. Edit User

Field Edit option

Summary The Summary tab opens by default, and is not editable. You can copy the User ARN.

Groups Groups tab consists of: Groups that are associated with the selected user. ADD GROUPS

To add one or more groups, select > ADD GROUPS. In the wizard that appears, select the group to add and then click SAVE.

REMOVE USER FROM GROUPS

Working with IAM accounts and users 33

Table 6. Edit User (continued)

Field Edit option

To remove a user from one or more groups, select one or more groups from the Group list and click REMOVE USER FROM GROUPS.

Permissions Permissions tab consists of: MANAGED POLICIES INLINE POLICY BOUNDARIES

MANAGED POLICIES tab is displayed by default. To attach a policy, select:

ATTACH POLICY > Copy permissions from user > User list > SAVE, or ATTACH POLICY > Policies > select one or more policies > SAVE

To detach a policy, select one or more policies > DETACH > SAVE INLINE POLICY tab allows you to:

ADD INLINY POLICY DETACH

BOUNDARIES tab allows you to: CHANGE REMOVE

Tags Tags tab consists of: ADD TAGS EDIT DELETE

Secret Key Secret Key tab consists of: ADD KEY REMOVE ACTIVATE DEACTIVATE

Delete an IAM user account

This task describes how to delete an IAM user account in ObjectScale.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Select an account from the account list and select Users. Select a user from the user list. The Summary tab opens by default.

4. Select Secret Key The user Secret Key tab appears and displays the keys of the user.

5. Select all the keys and click Remove.

6. Click < Users at the top of the user tab to return to the list of users.

NOTE: In order to delete a user, you should first delete the associated permissions and policies that are attached in

addition to deleting the secret key.

7. Select the IAM user account from the User list to remove and click DELETE.

34 Working with IAM accounts and users

Groups A Group is a collection of Users. You can use groups to specify permissions for a collection of users.

Use the following tasks to manage ObjectScale IAM groups.

New Group

This task describes how to add groups to an Account in ObjectScale.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Select an account from the account list and then select the Groups tab. The Groups list appears.

4. Click NEW GROUP. The NEW GROUP window opens.

5. Fill all the required fields in the NEW GROUP window.

Name Policies

6. Click SAVE. A New Group is created for the account.

Edit Group

This task describes how to edit groups of an Account in ObjectScale.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Select an account from the account list and then select the Groups tab. The Groups list appears.

4. Select a group, and click EDIT. The Summary tab opens, by default, and is not editable. The other two tabs are:

Users, and Permissions.

5. To add or remove users from the group, do the following: To add a user to the group, click Users > ADD USER > select one or more users > SAVE. To remove a user from the group, click Users > select one or more users > REMOVE > SAVE.

6. To edit permissions for the group, select Permissions.

Permissions for groups consists of: MANAGED POLICIES INLINE POLICIES

Working with IAM accounts and users 35

Delete Groups

This task describes how to delete groups from an Account in ObjectScale.

Prerequisites

Before you delete a group first remove all the users who are attached to the group, along with the permissions and policies.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Select an account from the account list and then select the Groups tab. The Groups list appears.

4. Select one more or more Groups, and click DELETE. A confirmation window opens that displays the selected Groups to be deleted.

5. Click YES. The selected Groups are deleted from the account.

Roles A role is similar to a user, in that it is an identity with permission policies that determine what the identity can and cannot do.

Instead of being uniquely associated with one person, a role is intended to be assumable by anyone who needs it. Also, a role does not have any credentials (password or access keys) associated with it. Instead, if a user is assigned to a role, access keys are created dynamically and provided to the user.

Use the following tasks to manage ObjectScale IAM roles.

New Role

This task describes how to add role to an Account in ObjectScale.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Select an account from the account list and then select the Roles tab. The Roles list appears.

4. Click NEW ROLE.

The New Role window opens.

36 Working with IAM accounts and users

Figure 5. New Role - General tab

General tab is selected by default.

5. Fill all the required fields in the General tab.

The General tab consists of: Name. Name of the new role. Description. Add details about the new role. Session Duration. The default selected value is one hour.

6. Click NEXT. The Trust tab opens.

7. Fill all the required fields in the Trust tab.

The Trust tab consists of: Set Effect, click Allow or Deny. Account. The Account tab is selected by default.

a. To Add Principal ARN, click ADD PRINCIPAL ARN and provide the Principal ARN value to the text field. b. To Add Service Principal, click the slider to enable. c. To go to the Permissions tab, click NEXT.

SAML2.0 Federation. a. From the drop-down menus, select:

SAML Provider Attribute Value Conditions, with Key, Condition, and Value.

b. To go to the Permissions (optional) tab, click NEXT.

8. Select policies to be associated with the new rule in the Permissions tab.

You can select one or more policies from the tab All policies tab. The All policies tab is selected by default. System Managed policies tab.

Working with IAM accounts and users 37

Customer Managed policies tab. , and click NEXT.

Optionally, you can add Permissions Boundary, by enabling the Permission Boundary slider and selecting the required policies.

9. Fill all the required fields in the Tags (optional) tab.

The Tags (optional) tab allows you to: Enter values in the Key and Value fields for a tag. To delete a tag, select the tag and click Delete. To add a tag, click ADD TAG, and enter the values in the Key and Value fields.

10. Click NEXT. The Review tab opens.

11. Review all the required fields in the Review tab.

12. Click SAVE. The New Role is saved with all the provided information and policies.

Edit Roles

This task describes how to edit roles in ObjectScale.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Select an account from the account list and then select the Roles tab. The Roles list appears.

4. Select a role from the list. The Roles details for the select role appear. Summary Trust Permissions Tags

5. To edit the fields in the Summary tab, click EDIT.

You cannot edit the Name field. You can only edit the Description, and Session Duration fields.

6. Click SAVE.

7. To edit the fields in the Trust tab, click Trust.

NOTE: The ACCOUNT tab is displayed by default.

The Trust tab consists of: ACCOUNT SAML 2.0 FEDERATION

Table 7. Account

Field Action

Effect To edit Effect: a. Click EDIT. b. Select Allow or Deny. c. Click SAVE.

Principal ARN. To add Principal ARN: a. Click ADD PRINCIPAL ARN.

38 Working with IAM accounts and users

Table 7. Account (continued)

Field Action

b. Add one or more principle ARN. c. Click SAVE.

To edit Principal ARN: a. Select a principal ARN. b. Click EDIT. c. Edit the principal ARN. d. Click SAVE.

To delete Principal ARN: a. Select a principal ARN. b. Click DELETE.

Service ARN To delete Service ARN: a. Select a Service ARN. b. Click DELETE.

Table 8. SAML 2.0 Federation

Field Action

SAML 2.0 FEDERATION To edit SAML 2.0 FEDERATION: a. Select a SAML Provider. b. Select an Attribute. c. Select a Value. d. Click SAVE.

To add a condition: a. Select a ADD CONDITION. b. Provide the condition information. c. Click SAVE.

To edit a condition: a. Select a condition from the table. b. Click EDIT. c. Modify the condition values. d. Click SAVE.

To delete a condition: a. Select a condition from the table. b. Click DELETE.

8. To edit the policies that are attached to the role, click Permissions. Select a policy from the MANAGED POLICIES, INLINE POLICIES, or BOUNDARY tab and click ATTACH POLICY or DETACH POLICY.

9. To edit the fields in the Tags tab, click Tags. You can ADD TAGS, EDIT, or DELETE from the selected role.

Delete Roles

This task describes how to delete roles in ObjectScale.

Prerequisites

Before you delete a role, first remove all the permissions and policies that are attached to it.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

Working with IAM accounts and users 39

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Select an account from the account list and then select the Roles tab. The Roles list appears.

4. Select one or more roles, and click DELETE. A confirmation window opens that displays the selected roles to be deleted.

5. Click YES. The selected roles are deleted from the account.

Policies IAM policies are documents in JSON format that define permissions for an operation regardless of the method that you use to perform the operation.

The table below describes the policy types, that are designed for use in ObjectScale.

Table 9. IAM Policies

Identity-based policies

Identity-based policies grant permissions to an IAM entity to control what actions an entity (users, groups of users, and roles) can perform, on which resources, and under what conditions.

In ObjectScale, resource-based policies are further categorized as:

ObjectScale managed policies Created and managed by ObjectScale. These policies cannot be modified or deleted.

Customer-managed policies Managed policies that users create and manage in account.

Inline policies Policies that are added to a single user, group, or role.

Resource-based policies

Attached inline policies to resources. Resource-based policies grant permissions to the principal that is specified in the policy. Principals can be in the same account as the resource or in other accounts.

In ObjectScale, resource-based policies are further categorized as:

S3 bucket policies IAM role trust policies

Permissions boundaries

Sets the maximum permissions that an identity-based policy can grant to an IAM entity (user or role). When you set a permissions boundary for an entity, the entity can perform only the actions that are allowed by both its identity-based policies and its permissions boundaries. Resource-based policies that specify the user or role as the principal are not limited by the permissions boundary. An explicit deny in any of these policies overrides the allow.

Session policies Session policies are advanced policies that you enter a parameter when you programmatically create a temporary session for a role. The permissions for a session are the intersection of the identity-based policies for the IAM entity (user or role) used to create the session and the session policies. Permissions can also come from a resource-based policy. An explicit deny in any of these policies overrides the allow.

Use the following tasks to manage ObjectScale IAM policies.

NOTE: Only customer-managed policy documents can be edited or deleted.

Create a new customer-managed policy

This task describes how to add policies to an Account in ObjectScale.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

40 Working with IAM accounts and users

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Select an account from the account list and then select the Policies tab. The Policies list appears.

4. Click NEW POLICY. The New Policy window opens.

5. Create the policy using the New Policy wizard.

a. On the General tab, add Name and Description for the new policy. b. On the Editor tab, click Visual or JSON to fill out the policy editor.

Click ADD POLICY STATEMENT to add additional statements.

Figure 6. New Policy - Editor tab

c. On the Review tab, verify that the previewed policy statement is accurate and then click SAVE.

New policy is created for the account.

Edit a customer-managed policy

This task describes to edit policies that are attached to an Account in ObjectScale.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

Working with IAM accounts and users 41

3. Select an account from the account list and then select the Policies tab. The Policies list appears.

4. Select the customer-managed policy to modify. The policy details are displayed, and the policy Summary tab is shown by default.

5. Edit the aspects of the customer-managed policy. To edit the policy permissions, select the Permissions tab.

On the Permissions tab, you can CLONE or DELETE policy statements. You can also edit the Service, Action, Resources, and Request Condition values for this policy.

Figure 7. Policy Permissions tab To edit the policy usage, select the Usage tab.

42 Working with IAM accounts and users

Figure 8. Policy Usage tab To manage the versions of the policy, select the Versions tab.

Figure 9. Policy Permissions tab

Delete a customer-managed policy

This task describes how to delete a customer-managed policy from an Account.

Prerequisites

To avoid a conflict from the deletion, you should first remove all the subordinate entities that are attached to the IAM-managed policy before deleting the policy.

Working with IAM accounts and users 43

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Select an account from the account list and then select the Policies tab. The Policies list appears.

4. Select one or more policies, and click DELETE. A confirmation window opens that displays the selected policies to be deleted.

5. Click YES. The selected policies are deleted from the account.

Actions in IAM Policy

This section describes all the supported actions in IAM policies that allow system account users or IAM users to perform operations.

Actions supported for system account user

Table 10. Account

Action Description Access Level

Resource Type (* required)

Condition Keys

account:* All account actions. N/A - -

account:CreateAccount Create account in ObjectScale instance.

Write - -

account:UpdateAccount Update account configuration. Write - -

account:ListAccounts List all accounts created in all ObjectScale instances.

List - -

account:GetAccount Retrieves information about the specified account.

Read - -

account:DeleteAccount Delete specified account. Write - -

account:AssociateAccou ntToObjectStore

Associate account to object store. Write - If required, can support below condition key . account:object StoreId

account:UnassociateAcc ountToObjectStore

Disassociate account from object store.

Write - -

Table 11. Grafana

Action Description Access Level

Resource Type (* required)

Condition Keys

grafana:* Grant all access for all object store operations.

* - -

44 Working with IAM accounts and users

Table 12. Object store

Action Description Access Level

Resource Type (* required)

Condition Keys

objectstore:* Grant access for all object store operations.

* - -

objectstore:Get* Grant read access to object store. Read - -

objectstore:Write* Grant write access to object store. Write - -

Table 13. Influxdb

Action Description Access Level

Resource Type (* required)

Condition Keys

influxdb:* Grant all influxdb operations. * - -

influxdb:Get* Grant influxdb read. Read - -

influxdb:Write* Grant influxdb write. Write - -

Table 14. Alert

Action Description Access Level

Resource Type (* required)

Condition Keys

Alert:* Allows system account user to perform all alert operations.

* - -

Actions supported for IAM user

Table 15. IAM entity management

Action Description Access Level

Resource Type (* required)

Condition Keys

iam:AddUserToGroup Adds an IAM user to the specified IAM group.

Write group* -

iam:AttachGroupPolicy Attaches a specified managed policy to the specified IAM group.

Permissions management

group* iam:PolicyARN

iam:AttachRolePolicy Attaches a specified managed policy to the specified IAM role.

Permissions management

role* iam:PolicyARN iam:Permission sBoundaryiam:R esourceTag/$ {TagKey}

iam:AttachUserPolicy Attaches a specified managed policy to the specified IAM user.

Permissions management

user* iam:PolicyARN iam:Permission sBoundary iam:ResourceTa g/${TagKey}

iam:CreateAccessKey Creates a new secret access credential for specified IAM user.

Write user* -

iam:CreateGroup Creates a IAM group in namespace. Write group* -

iam:CreatePolicy Creates a new managed policy in namespace.

Permissions management

policy* -

iam:CreatePolicyVersio n

Creates a version of the specified managed policy in namespace.

Permissions management

policy* -

Working with IAM accounts and users 45

Table 15. IAM entity management (continued)

Action Description Access Level

Resource Type (* required)

Condition Keys

iam:CreateRole Creates a IAM role in namespace. Write role* iam:Permission sBoundary

iam:CreateSAMLProvider Creates a SAML 2.0 identity provider (IdP) in namespace.

Write saml-provider* -

iam:CreateUser Creates an IAM user in namespace. Write user* iam:Permission sBoundary

iam:DeleteAccessKey Deletes the specified access key credentials that are associated with the specified IAM user.

Write user* -

iam:DeleteGroup Deletes the specified IAM group from namespace.

Write group* -

iam:DeleteGroupPolicy Deletes the specified inline policy from its group.

Permissions management

group* -

iam:DeletePolicy Deletes the specified managed policy. Permissions management

policy* iam:PolicyARN

iam:DeletePolicyVersio n

Deletes the specified version from the managed policy.

Permissions management

policy* -

iam:DeleteRole Grants permission to delete the specified role.

Write role* -

iam:DeleteRolePermissi onsBoundary

Deletes the permissions boundary for the specified IAM role.

Permissions management

role* iam:Permission sBoundaryiam:R esourceTag/$ {TagKey}

iam:DeleteRolePolicy Deletes the specified inline policy from its role.

Permissions management

role* iam:Permission sBoundary

iam:DeleteSAMLProvider Deletes a specified SAML provider. Write saml-provider* -

iam:DeleteUser Deletes the specified IAM user from namespace.

Write user* iam:ResourceTa g/${TagKey}

iam:DeleteUserPermissi onsBoundary

Deletes the permissions boundary for the specified IAM user.

Permissions management

user* iam:Permission sBoundaryiam:R esourceTag/$ {TagKey}

iam:DeleteUserPolicy Deletes the specified inline policy from its user.

Permissions management

user* iam:Permission sBoundary iam:ResourceTa g/${TagKey}

iam:DetachGroupPolicy Detach a specified managed policy from the specified IAM group.

Permissions management

group* -

iam:DetachRolePolicy Detach a specified managed policy from the specified IAM role.

Permissions management

role* iam:PolicyARN iam:Permission sBoundaryiam:R esourceTag/$ {TagKey}

iam:DetachUserPolicy Detach a specified managed policy from the specified IAM user.

Permissions management

user* iam:PolicyARN iam:Permission sBoundaryiam:R esourceTag/$ {TagKey}

46 Working with IAM accounts and users

Table 15. IAM entity management (continued)

Action Description Access Level

Resource Type (* required)

Condition Keys

iam:GetAccessKeyLastUs ed

Retrieves best effort information about when specified access key was last used.

Read user* -

iam:GetContextKeysForC ustomPolicy

Retrieves list of all the context keys that are referenced in the input policies.

Read - -

iam:GetContextKeysForP rincipalPolicy

Detach a specified managed policy from the specified IAM entity.

Read user, group, role

-

iam:GetGroup Returns a list of IAM users that are in the specified IAM group. You can paginate the results using the MaxItems and Marker parameters.

Read group* -

iam:GetGroupPolicy Gets the specified inline policy document from the specified IAM group.

Read group* -

iam:GetPolicy Retrieve information about the specified managed policy.

Read policy* -

iam:GetPolicyVersion Retrieve information about a version of the specified managed policy.

Read policy* -

iam:GetRole Retrieves information about the specified role.

Read role* iam:ResourceTa g/${TagKey}

iam:GetPolicy Retrieves information about specified managed policy.

Read policy* -

iam:GetPolicyVersion Retrieves information about specified version of the managed policy.

Read policy* -

iam:GetRolePolicy Retrieves the specified inline policy document that is embedded with the specified IAM role.

Read role* iam:ResourceTa g/${TagKey}

iam:GetSAMLProvider Retrieves the SAML provider metadata document that is associated with the IAM SAML provider resource.

Read saml-provider* -

iam:GetUser Retrieves information about the specified IAM user.

Read user* iam:ResourceTa g/${TagKey}

iam:GetUserPolicy Retrieves the specified inline policy document of the specified IAM user.

Read user* iam:ResourceTa g/${TagKey}

iam:ListAccessKeys Lists information about the access key IDs that are associated with the specified IAM user.

List user* -

iam:ListAttachedGroupP olicies

Lists all managed policies that are attached to the specified IAM group.

List group* -

iam:ListAttachedRolePo licies

Lists all managed policies that are attached to the specified IAM role.

List role* iam:ResourceTa g/${TagKey}

iam:ListAttachedUserPo licies

Lists all managed policies that are attached to the specified IAM user.

List user* iam:ResourceTa g/${TagKey}

iam:ListEntitiesForPol icy

Lists all entities (IAM users, groups, and roles) that are attached to the specified managed policy.

List policy* -

Working with IAM accounts and users 47

Table 15. IAM entity management (continued)

Action Description Access Level

Resource Type (* required)

Condition Keys

iam:ListGroupPolicies List the names of the inline policies that are in the specified IAM group.

List group* -

iam:ListGroups List the IAM groups that have the specified path prefix.

List - -

iam:ListGroupsForUser List the IAM groups that the provided IAM user belongs to.

List user* iam:ResourceTa g/${TagKey}

iam:ListPolicies Lists all managed policies that are associated with the namespace.

List - -

iam:ListPolicyVersions Lists information about the versions of the requested managed policy.

List policy* -

iam:ListRolePolicies Lists the names of the inline policies that are in the specified IAM role.

List role* iam:ResourceTa g/${TagKey}

iam:ListRoles Lists the IAM roles that have the specified path prefix.

List - -

iam:ListRoleTags Lists the tags that are attached to the specified role.

List role* iam:ResourceTa g/${TagKey}

iam:ListSAMLProviders Lists the SAML providers that are in the namespace.

List - -

iam:ListUserPolicies Lists the names of the inline policies that are in the specified IAM user.

List user* iam:ResourceTa g/${TagKey}

iam:ListUsers Lists the IAM users that have the specified path prefix.

List - -

iam:ListUserTags Lists the tags that are attached to the specified user.

List user* iam:ResourceTa g/${TagKey}

iam:PutGroupPolicy Adds or updates an inline policy document to the specified IAM group.

Permissions management

group* -

iam:PutRolePermissions Boundary

Sets or updates the provided managed policy as the roles permissions boundary.

Permissions management

role* iam:Permission sBoundaryiam:R esourceTag/$ {TagKey}

iam:PutRolePolicy Adds or updates an inline policy document to the specified IAM role.

Permissions management

role* iam:Permission sBoundaryiam:R esourceTag/$ {TagKey}

iam:PutUserPermissions Boundary

Sets or updates the provided managed policy as the permissions boundary of a user.

Permissions management

user* iam:Permission sBoundaryiam:R esourceTag/$ {TagKey}

iam:PutUserPolicy Adds or updates an inline policy document to the specified IAM user.

Permissions management

user* iam:Permission sBoundaryiam:R esourceTag/$ {TagKey}

iam:RemoveUserFromGrou p

Removes an IAM user from the specified group.

Write group* -

iam:SetDefaultPolicyVe rsion

Sets the specified version of the policy as default.

Permissions management

policy* -

48 Working with IAM accounts and users

Table 15. IAM entity management (continued)

Action Description Access Level

Resource Type (* required)

Condition Keys

iam:SimulateCustomPoli cy

Simulates how a set of IAM policies and optionally a resource-based policy works with a list of API operations and ObjectScale resources to determine the effective permissions of the policy.

Read - -

iam:SimulatePrincipalP olicy

Simulates how a set of IAM policies that are attached to an IAM entity (user, group, or role) works with a list of API operations and ObjectScale resources to determine the effective permissions of the policy.

Read user, group, role

-

iam:TagRole Adds tags to an IAM role. Tagging role* -

iam:TagUser Adds tags to an IAM user. Tagging user* -

iam:UntagRole Removes tags from specified IAM role. Tagging role* iam:ResourceTa g/${TagKey}

iam:UntagUser Removes tags from specified IAM user.

Tagging user* iam:ResourceTa g/${TagKey}

iam:UpdateAccessKey Updates the status of specified access keys as Active or Inactive.

Write user* -

iam:UpdateAssumeRolePo licy

Updates the policy that grants an IAM entity permission to assume a role.

Permissions management

role* iam:ResourceTa g/${TagKey}

iam:UpdateRole Updates the description or maximum session duration setting of an IAM role.

Write role* iam:ResourceTa g/${TagKey}

iam:UpdateSAMLProvider Updates the metadata document for an existing SAML provider.

Write saml-provider* -

Table 16. S3

Action Description Access Level

Resource Type (* required)

Condition Keys

New operations supported by S3 service:

s3:GetReplicationConfi guration

Grants permission to get the replication configuration information set on an amazon S3 bucket.

Read bucket* s3:authType s3:signature version s3:x-amz- content- sha256

s3:PutReplicationConfi guration

Grants permission to create a replication configuration or replace an existing one.

Write bucket* s3:authType s3:signature version s3:x-amz- content- sha256

s3:DeleteReplicationCo nfiguration

Grants permission to delete a replication configuration.

Write bucket* -

Working with IAM accounts and users 49

Table 16. S3 (continued)

Action Description Access Level

Resource Type (* required)

Condition Keys

s3:GetBucketObjectLock Configuration

Grants permission to get the object lock configuration of an amazon S3 bucket.

Read bucket* s3:authType s3:signature version

s3:PUTBucketObjectLock Configuration

Grants permission to get the object lock configuration of an amazon S3 bucket.

Write bucket* s3:authType s3:signature version

s3:GetObjectLegalHold Grants permission to get the current legal hold status of an object.

Read object* s3:authType s3:signature version s3:x-amz- content- sha256

s3:PutObjectLegalHold Grants permission to apply a legal hold configuration to a specified object.

Write object* s3:authType s3:signature version s3:x-amz- content- sha256 s3:object- lock-legal- hold

s3:GetObjectRetention Grants permission to retrieve the retention settings for an object.

Read object* s3:authType s3:signature version s3:x-amz- content- sha256

s3:PutObjectRetention Grants permission to place an object retention configuration on an object.

Write object* s3:authType s3:signature version s3:x-amz- content- sha256 s3:object- lock-mode s3:object- lock-retain- until-date s3:object- lock- remaining- retention- days

s3:BypassGovernanceRet ention

Grants permission to allow circumvention of governance-mode object retention settings.

Permission Management

object* s3:RequestOb jectTag/ s3:RequestOb jectTagKeys s3:authType s3:signature version

50 Working with IAM accounts and users

Table 16. S3 (continued)

Action Description Access Level

Resource Type (* required)

Condition Keys

s3:x-amz- acl s3:x- amz-content- sha256 s3:x- amz-copy- source s3:x- amz-grant- full- control s3:x-amz- grant-read s3:x-amz- grant-read- acp s3:x- amz-grant- write s3:x- amz-grant- write-acp s3:x-amz- metadata- directive s3:x-amz- server-side- encryption s3:x-amz- storage- class s3:object- lock-mode s3:object- lock-retain- until-date s3:object- lock- remaining- retention- days s3:object- lock-legal- hold

Existing S3 operations supported by S3 service:

s3:AbortMultipartUploa d

Grants permission to cancel a multipart upload.

Write object* s3:authType s3:signature version s3:x-amz- content- sha256

s3:CreateBucket Grants permission to create a bucket. Write bucket* s3:authType s3:signature version s3:x-amz-acl s3:x-amz- content- sha256 s3:x-amz- grant-full- control s3:x-amz- grant-read s3:x-amz- grant-read-

Working with IAM accounts and users 51

Table 16. S3 (continued)

Action Description Access Level

Resource Type (* required)

Condition Keys

acp s3:x-amz- grant-write s3:x-amz- grant-write- acp

s3:DeleteBucket Grants permission to delete the bucket named in the URI.

Write bucket* s3:authType s3:signature version s3:x-amz- content- sha256

s3:DeleteBucketPolicy Grants permission to delete policy on a specified bucket.

Permission Management

bucket* s3:authType s3:signature version s3:x-amz- content- sha256

s3:DeleteObject Grants permission to remove the null version of an object and insert a delete marker, which becomes the current version of the object.

Write object* s3:authType s3:signature version s3:x-amz- content- sha256

s3:DeleteObjectTagging Grants permission to use the tagging subresource to remove the entire tag set from the specified object.

Tagging object* s3:ExistingO bjectTag/ s3:authType s3:signature version s3:x-amz- content- sha256

s3:DeleteObjectVersion Grants permission to remove a specific version of an object.

Write object* s3:authType s3:signature version s3:versionid s3:x-amz- content- sha256

s3:DeleteObjectVersion Tagging

Grants permission to remove the entire tag set for a specific version of the object.

Tagging object* s3:ExistingO bjectTag/ s3:authType s3:signature version s3:versionid s3:x-amz- content- sha256

52 Working with IAM accounts and users

Table 16. S3 (continued)

Action Description Access Level

Resource Type (* required)

Condition Keys

s3:GetBucketAcl Grants permission to use the ACL subresource to return the access control list (ACL) of an Amazon S3 bucket.

Read bucket* s3:authType s3:signature version s3:x-amz- content- sha256

s3:GetBucketCORS Grants permission to return the CORS configuration information set for an Amazon S3 bucket.

Read bucket* s3:authType s3:signature version s3:x-amz- content- sha256

s3:GetBucketPolicy Grants permission to return the policy of the specified bucket.

Read bucket* s3:authType s3:signature version s3:x-amz- content- sha256

s3:GetBucketTagging Grants permission to return the tag set associated with an Amazon S3 bucket.

Read bucket* s3:authType s3:signature version s3:x-amz- content- sha256

s3:GetBucketVersioning Grants permission to return the versioning state of an Amazon S3 bucket.

Read bucket* s3:authType s3:signature version s3:x-amz- content- sha256

s3:GetLifecycleConfigu ration

Grants permission to return the life- cycle configuration information set on an Amazon S3 bucket.

Read bucket* s3:authType s3:signature version s3:x-amz- content- sha256

s3:GetObject Grants permission to retrieve objects from Amazon S3.

Read object* s3:ExistingO bjectTag/ s3:authType s3:signature version s3:x-amz- content- sha256

s3:GetObjectAcl Grants permission to return the access control list (ACL) of an object.

Read object* s3:ExistingO bjectTag/

Working with IAM accounts and users 53

Table 16. S3 (continued)

Action Description Access Level

Resource Type (* required)

Condition Keys

s3:authType s3:signature version s3:x-amz- content- sha256

s3:GetObjectTagging Grants permission to return the tag set of an object.

Read object* s3:ExistingO bjectTag/ s3:authType s3:signature version s3:x-amz- content- sha256

s3:GetObjectVersion Grants permission to retrieve a specific version of an object.

Read object* s3:ExistingO bjectTag/ s3:authType s3:signature version s3:versionid s3:x-amz- content- sha256

s3:GetObjectVersionAcl Grants permission to return the access control list (ACL) of a specific object version.

Read object* s3:ExistingO bjectTag/ s3:authType s3:signature version s3:versionid s3:x-amz- content- sha256

s3:GetObjectVersionTag ging

Grants permission to return the tag set for a specific version of the object.

Read object* s3:ExistingO bjectTag/ s3:authType s3:signature version s3:versionid s3:x-amz- content- sha256

s3:ListAllMyBuckets Grants permission to list all buckets owned by the authenticated sender of the request.

List s3:authType s3:signature version s3:x-amz- content- sha256

54 Working with IAM accounts and users

Table 16. S3 (continued)

Action Description Access Level

Resource Type (* required)

Condition Keys

s3:ListBucket Grants permission to list some or all the objects in an Amazon S3 bucket (up to 1000).

List bucket* s3:authType s3:delimiter s3:max-keys s3:prefix s3:signature version s3:x-amz- content- sha256

s3:ListBucketMultipart Uploads

Grants permission to list in-progress multipart uploads.

Read bucket* s3:authType s3:signature version s3:x-amz- content- sha256

s3:ListBucketVersions Grants permission to list metadata about all the versions of objects in an Amazon S3 bucket.

Read bucket* s3:authType s3:delimiter s3:max-keys s3:prefix s3:signature version s3:x-amz- content- sha256

s3:ListMultipartUpload Parts

Grants permission to list the parts that have been uploaded for a specific multipart upload.

Read object* s3:authType s3:signature version s3:x-amz- content- sha256

s3:PutBucketAcl Grants permission to set the permissions on an existing bucket using access control lists (ACLs).

Permission Management

bucket* s3:authType s3:signature version s3:x-amz-acl s3:x-amz- content- sha256 s3:x-amz- grant-full- control s3:x-amz- grant-read s3:x-amz- grant-read- acp s3:x-amz- grant-write s3:x-amz- grant-write- acp

s3:PutBucketCORS Grants permission to set the CORS configuration for an Amazon S3 bucket.

Write bucket* s3:authType s3:signature version

Working with IAM accounts and users 55

Table 16. S3 (continued)

Action Description Access Level

Resource Type (* required)

Condition Keys

s3:x-amz- content- sha256

s3:PutBucketPolicy Grants permission to add or replace a bucket policy on a bucket.

Permission Management

bucket* s3:authType s3:signature version s3:x-amz- content- sha256

s3:PutBucketTagging Grants permission to add a set of tags to an existing Amazon S3 bucket.

Tagging bucket* s3:authType s3:signature version s3:x-amz- content- sha256

s3:PutBucketVersioning Grants permission to set the versioning state of an existing Amazon S3 bucket.

Write bucket* s3:authType s3:signature version s3:x-amz- content- sha256

s3:PutLifecycleConfigu ration

Grants permission to create a life- cycle configuration for the bucket or replace an existing life-cycle configuration.

Write bucket* s3:authType s3:signature version s3:x-amz- content- sha256

s3:PutObject Grants permission to add an object to a bucket.

Write object* s3:RequestOb jectTag/ s3:RequestOb jectTagKeys s3:authType s3:signature version s3:x-amz-acl s3:x-amz- content- sha256 s3:x-amz- copy-source s3:x-amz- grant-full- control s3:x-amz- grant-read s3:x-amz- grant-read- acp s3:x-amz- grant-write s3:x-amz- grant-write- acp

56 Working with IAM accounts and users

Table 16. S3 (continued)

Action Description Access Level

Resource Type (* required)

Condition Keys

s3:x-amz- metadata- directive s3:x-amz- server-side- encryption s3:x-amz- server-side- encryption- aws-kms-key- id s3:x-amz- storage- class s3:object- lock-mode s3:object- lock-retain- until-date s3:object- lock- remaining- retention- days s3:object- lock-legal- hold

s3:PutObjectAcl Grants permission to set the access control list (ACL) permission for an object that exists in a bucket.

Permission Management

object* s3:ExistingO bjectTag/ s3:authType s3:signature version s3:x-amz-acl s3:x-amz- content- sha256 s3:x-amz- grant-full- control s3:x-amz- grant-read s3:x-amz- grant-read- acp s3:x-amz- grant-write s3:x-amz- grant-write- acp s3:x-amz- storage- class

s3:PutObjectTagging Grants permission to set the supplied tag-set to an object that exists in a bucket.

Tagging object* s3:ExistingO bjectTag/ s3:RequestOb jectTag/ s3:RequestOb jectTagKeys

Working with IAM accounts and users 57

Table 16. S3 (continued)

Action Description Access Level

Resource Type (* required)

Condition Keys

s3:authType s3:signature version s3:x-amz- content- sha256

s3:PutObjectVersionAcl Grants permission to use the ACL subresource to set the access control list (ACL) permissions for an object that exists in a bucket.

Permission Management

object* s3:ExistingO bjectTag/ s3:authType s3:signature version s3:versionid s3:x-amz-acl s3:x-amz- content- sha256 s3:x-amz- grant-full- control s3:x-amz- grant-read s3:x-amz- grant-read- acp s3:x-amz- grant-write s3:x-amz- grant-write- acp s3:x-amz- storage- class

s3:PutObjectVersionTag ging

Grants permission to set the supplied tag-set for a specific version of an object.

Tagging object* s3:ExistingO bjectTag/ s3:RequestOb jectTag/ s3:RequestOb jectTagKeys s3:authType s3:signature version s3:versionid s3:x-amz- content- sha256

s3:ReplicationInfo Grants permission to retrieve and object replication status.

Read object* -

Table 17. STS

Action Description Access Level

Resource Type (* required)

Condition Keys

sts:AssumeRole Returns a set of temporary security credentials that you can use to access

Write role* aws:RequestT ag/${TagKey}

58 Working with IAM accounts and users

Table 17. STS (continued)

Action Description Access Level

Resource Type (* required)

Condition Keys

ObjectScale resources that you might not have access to.

aws:TagKeys aws:Principa lTag/$ {TagKey}

sts:AssumeRoleWithSAML Returns a set of temporary security credentials for users who have been authenticated using a SAML authentication response.

Write role* aws:RequestT ag/${TagKey} aws:TagKeys aws:Principa lTag/$ {TagKey} saml:iss saml:aud saml:sub saml:sub_typ e saml:edupers onorgdn saml:namequa lifier

Table 18. DCM

Action Description Access Level

Resource Type (* required)

Condition Keys

dcm:GetWebhookConfigur ation

Retrieves a webhook configuration to which notifications can be published.

Read webhook* Dependent actions

dcm:CreateWebhookConfi guration

Creates a webhook configuration to which notifications can be published.

Write webhook* Dependent actions

dcm:DeleteWebhookConfi guration

Deletes a webhook configuration. Write webhook* Dependent actions

dcm:ListWebhookConfigu rations

Returns a list of webhook configurations of the requester. Each call returns a limited list of configurations, up to 100.

List webhook* Dependent actions

Principal types in IAM Policies

This section lists all the supported principal types in IAM policies that allow system account users or IAM users to perform operations.

NOTE: Principal element is not used in IAM identity-based policies. You can use principal elements in trust policies for IAM

roles and in resource-based policies(bucket policy and trust policy).

Table 19. Principal Types in IAM Policies

Action Format Description

AWS account and root user "Principal": { "AWS": "urn:ecs:iam:: :root" }

-

IAM users "Principal": { "AWS": "urn:ecs:iam:: :user/ " } "Principal":

Replicated IAM user

Local IAM user

Working with IAM accounts and users 59

Table 19. Principal Types in IAM Policies (continued)

Action Format Description

{ "AWS": "urn:flexlocal:iam:: :user/ " }

Federated users (using SAML federation) "Principal": {

"Federated": "urn:ecs:iam:: :saml- provider/ " }

-

IAM roles "Principal": { "AWS": "urn:ecs:iam:: :role/ " } "Principal": { "AWS": "urn:flexlocal:iam:: :role/ " }

Replicated IAM role

Local IAM role

Assumed-role sessions "Principal": { "AWS": "urn:ecs:sts:: :assumed-role/ / " "Principal": { "AWS": "urn:flexlocal:sts:: :assumed- role/ / "

-

Services "Principal": {"Service": " "} -

Anonymous users "Principal" : { "AWS" : "*" } -

Identity Provider IAM in ObjectScale allows you to:

Use the following tasks to manage identity providers.

New Identity Provider

This task describes how to add Identity Providers to an Account in ObjectScale.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Select an account from the account list and then select the Identity Provider tab. The Identity Provider list appears.

4. Click New Identity Provider. The New Identity Provider window opens.

5. Fill all the required fields in the New Identity Provider window.

Name Type Metadata Provider

6. Click SAVE. A New Identity Provider is created for the account.

60 Working with IAM accounts and users

Edit Identity Providers

This task describes how to edit Identity Providers.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Select an account from the account list and then select the Identity Provider tab. The Identity Provider list appears.

4. Select an identity provider from the list and click Edit Identity Provider. The Edit Identity Provider window opens.

5. Edit the fields in the Edit Identity Provider window.

Name is not editable. Type Metadata Provider

6. Click SAVE. The Identity Provider details are updated for the account.

Delete Identity Providers

This task describes how to delete Identity Providers.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Select an account from the account list and then select the Identity Provider tab. The Identity Provider list appears.

4. Selects one or more Identity Providers, and click Delete Identity Provider. A confirmation window opens that displays the selected Identity Providers to be deleted.

5. Click YES. The selected Identity Providers are deleted from the account.

Root Access Keys Use the following tasks to manage ObjectScale IAM root access key(s).

Create a Root Access Key

This task describes how to generate a new key for an Account in ObjectScale.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts.

Working with IAM accounts and users 61

For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts.

For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Select an account from the account list and then select the Root Access Key tab. The Root Access Key list appears.

4. Click ADD KEY. The Add Secret Key window opens.

5. Click GENERATE.

6. Review the new secret key.

NOTE: This is the only time that the secret access keys can be viewed or downloaded. You cannot recover them later.

However, you can create access keys at any time.

Access Key ID

Access Secret Key - Optionally click Click DOWNLOAD

7. Click OK and optionally view the new key in the Root Access Key table.

Manage existing Root Access Keys

This task describes how to activate, deactivate, or remove a key in an Account in ObjectScale.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Select an account from the account list and then select the Root Access Key tab. The Root Access Key list appears.

4. Select the Root Access Keys to activate, deactivate, or remove. Click DEACTIVATE and then in the Deactivate access key(s) window and confirm the keys to deactivate and click

Yes.

If these are not the correct keys to deactivate, click No.

Click ACTIVATE and then in the Activate access key(s) window and confirm the keys to activate and click Yes.

If these are not the correct keys to activate, click No.

Click REMOVE and then in the Remove access key(s) window and confirm the keys to activate and click Yes.

If these are not the correct keys to remove, click No.

Notification Destinations Use the following tasks to manage notification destinations.

Create a notification destination

This task describes how to add a notification destination to an Account in ObjectScale.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

62 Working with IAM accounts and users

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Click an account from the account list and then click the Notification Destination tab. The Notification Destination list appears.

4. Click New Notification Destination. The New Notification Destination window opens.

5. Complete the required fields to create a notification destination:

Option Description

Name Type the name of the new destination.

Comment Type the description of the destination.

Endpoint Type the valid URL for the destination endpoint.

Authentication Token Type the authentication token value for the endpoint.

Backup Limit Select a backup limit for the first 100 destinations.

6. Click SAVE.

Edit a notification destination

This task describes how to edit an existing notification destination in an Account in ObjectScale.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Click an account from the account list and then click the Notification Destination tab. The Notification Destination list appears.

4. Select a notification destination from the list and then select the Edit tab. The Edit window appears.

5. Modify a destination value to modify the notification destination:

Option Description

Name Type the name of the new destination.

Comment Type the description of the destination.

Endpoint Type the valid URL for the destination endpoint.

Authentication Token Type the authentication token value for the endpoint.

Backup Limit Select a backup limit for the first 100 destinations.

6. Click SAVE.

Working with IAM accounts and users 63

Delete a notification destination

This task describes how to delete an existing notification destination in an Account in ObjectScale.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Accounts. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Accounts. For ObjectScale Portal UI: Click the Accounts tab. A user can see a list of Accounts that the user is authorized to view.

3. Click an account from the account list and then click the Notification Destination tab. The Notification Destination list appears.

4. Select a notification destination from the list and then select the Delete tab. The Delete window appears.

5. Confirm that the correct notification destination is selected and click OK.

Metrics The Metrics tab displays Account metrics on a Grafana UI.

For details about ObjectScale and object store metrics dashboards, see Viewing ObjectScale and object store metrics.

Also, see Grafana for basic details of navigation in Grafana dashboards.

64 Working with IAM accounts and users

Working with object stores This chapter contains:

Topics:

About ObjectScale object stores Create an object store Associate an IAM Account with an object store View object store summary View S3 and management certificate properties Edit an object store Set capacity alerts for an object store Delete an object store

About ObjectScale object stores ObjectScale introduces the new logical construct that is called the object store. Object stores are a discrete storage system with an individualized life cycle and are Kubernetes (k8s) applications that are deployed by ObjectScale.

One ObjectScale instance can contain multiple object stores. Object stores are created, updated, and deleted independently from all other object stores managed by ObjectScale. As you manage the object store though its life cycle, ObjectScale interacts with the underlying Kubernetes infrastructure as needed. Allowing Kubernetes to handle the necessary changes to the cluster resources, such as storage, CPU, and other resources.

You must associate an IAM account with an object store in order to allow users within the IAM account to manage aspects of the object store. When an IAM account is added to an object store, that account becomes a tenant within that object store. A tenant is a logical construct resulting from the binding of an IAM account to an object store.

After associating the IAM account with the object store, IAM accounts associated with the object store can create new buckets in the object store. these buckets are owned by the account creating the bucket. This is similar to AWS S3. the tenants can be used to specify quota restrictions for that account in that object store. you can set specific compliance settings. you can set specific retention policies.

The size of the persistent volumes (PVs) which are bound to the storage server (SS) pods in an object store represent the persistent storage capacity allocated for raw user data. An object store with three 200GB SS pods provides 600GB raw disk space. ObjectScale joins the persistent volumes and pods to hosts to protect data using erasure coding. Each object store has a maximum of one SS pod per k8s worker node.

To simplify the creation of object stores of the correct size and resource profile, ObjectScale implements a workload sizer tool within the workflows that are used to create or modify an object store. Administrators and end users, with the appropriate permissions, can choose the correct level of resources for the object store based on the expected workloads that the object store will handle. ObjectScale then dynamically calculates the object store requirements as you enter the required values for the object store. Supply the initial object store capacity and the expected growth rate to allow the sizing tool to correctly calculate the capacity requirements.

To support the workload inputs provided, ObjectScale then determines:

The number of replicas in the object store. PV size necessary to meet the storage needs for the life of the object store. Any additional capacity must cover overhead, such as metadata and data protection.

NOTE: If you plan on using vSAN Direct as the User Storage Class and/or Stream Storage Class, refer to Create a

custom vSAN SNA striped policy for the object store namespace. This custom policy allows ObjectScale to rebuild the

pods on the node, with the recovery process, even when one or two disks in a node are offline.

4

Working with object stores 65

Figure 10. New Object Store

In addition to the number and size of storage server instances that are required for an object store, ObjectScale also determines the size and quantity of all components that make up the object store. ObjectScale uses performance profiles to size object store resources. This release of ObjectScale includes the large performance profile.

NOTE: In certain circumstances ObjectScale will be unable to create the requested object store if certain pods are unable

to start. In these cases, ObjectScale will create an alert in the Health > Issues tab that provides the specific details on

which pod and what resource was not available. Use these details in the issue alert to resolve the underlying issue or cancel

the operation and remove any partial pod created prior to failure.

Object store naming conventions

This topic details the rules that apply to the naming of ObjectScale object stores.

Object store naming

The following rules apply to the naming of object stores in ObjectScale:

Object store names are required to be between three and 31 characters in length. Object store names can consist only of lowercase letters, numbers and hyphens (-). Object store names must begin with an alphabetic character, and end with an alphanumeric character. Object store names should be unique. Do not use the same name for two or more object stores within different namespaces

in the ObjectScale instance.

66 Working with object stores

Group label naming

Use this value to place a logical grouping construct on your object store. For example, "development" can be used to filter all your non-production object stores. The following rules apply to the group label in ObjectScale:

Group can be up to 63 characters long. Group can consist only of alphanumeric characters, hypen (-), underscore ( _ ), and dot (.) Group must begin and end with an alphanumeric character.

Create an object store

About this task

Each object store is a unique and independent storage system with an individualized lifecycle. One or more object stores are deployed by each ObjectScale instance. Object stores are created, updated, and deleted independently from all other object stores managed by the shared ObjectScale instance.

The New Object Store wizard is used to set the initial object store resource requirements based on the information collected to satisfy the demands specified by the administrator.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Select the namespace from the namespace drop-down on the top right corner of the ObjectScale UI.

NOTE: For vSphere, Dell EMC recommends that you use a namespace other than the namespace auto-created by

ObjectScale where the ObjectScale Operator is running.

4. Click New Object Store. The New Object Store wizard appears.

5. In the General configuration page complete the required fields.

Option Description

Name Enter the Name for the object store. You must verify the name by clicking the VERIFY button next to the name.

Namespace Verify that you are creating the object store in the correct namespace.

Version Select the Version.

Group Enter the Group value(s) to apply to the object store. Use this value to logically group the object stores and to filter the object stores with the group.

The Group field is optional.

My Templates If previously created, select a template to populate the configuration values for the new object store.

The My Templates field is optional.

Data protection type

ObjectScale is the only supported data protection type.

User Storage Class

Select the storage class to be used to provision the user storage for object store and bucket metadata. On vSphere, ObjectScale does not have visibility into the physical disks in VSAN SNA mode.

Therefore, tolerance to multiple physical disk failures is not considered for vSAN SNA. Failure tolerance will be at vSAN-SNA volume layer.

On vSphere, if you are selecting sna for the user storage class, then a custom striped policy is recommended for better performance. Follow the steps in "Create a custom vSAN SNA striped policy

Working with object stores 67

Option Description

for the object store namespace" in the ObjectScale Administration Guide for details on setting up the custom policy.

System Storage Class

Select the storage class to be used to provision the management service storage for metadata, service registration, and metrics data.

On OpenShift, it is highly recommended to use csi-baremetal-sc-ssdlvg storage class for any production object stores.

Stream Storage Class

Select the storage class to be used to provision the object store notification service storage for stream index, journal and ledger data. For OpenShift, it is highly recommended to use csi-baremetal-sc-ssdlvg storage class for

any production object stores. If there is no available storage of this type, use csi-baremetal-sc- hddlvg storage class.

For vSphere, ObjectScale does not have visibility into the physical disks in vSAN SNA mode. Therefore, tolerance to multiple physical disk failures is not considered for vSAN SNA. Failure tolerance will be at vSAN-SNA volume layer.

6. Click Quick or Advanced to continue with the creation of the new object store. Click Quick and go to step 10. Quick allows ObjectScale to make workload selections and jumps directly to the Storage

And Review section of the wizard. Click Advanced and go through the next steps to manually create the object store.

7. Complete the Topology configuration page.

Review the available resources and select the desired topology scheduling to apply.

To enable node exclusion using topology labels, select Advanced and select the nodes to exclude while creating the object store. You can filter the nodes by selecting the Source, Key, and Value of the desired nodes to exclude.

8. In the Storage configuration page, set the Configuration Type and click Next to continue. Select Automatic Layout to set the Usable Capacity and optionally expand and review the Details of the layout. Select Advanced Storage Options to define the custom sizing based on the capacity and performance required for the

object workloads. Set the values for Protection Scheme and Years of Growth. Optionally, select CUSTOMIZE to configure the Capacity per Storage Server Replica, Storage Server Replica

Count, and Volumes per Storage Server Replica.

NOTE: For vSphere, Dell EMC recommends the use of a minimum of 10 volumes per storage server replica for better

performance if vSAN-SNA type of storage is chosen for storing user data in General page.

Set the workloads for the custom storage tiers by selecting the Workloads tab and providing the Usable Capacity, Object Size, and Growth Rate for the object workload.

Click ADD WORKLOAD to create additional workload definitions.

9. Optional: Complete the Connectivity configuration page and click Next. Select Automatic Network Configuration to allow ObjectScale to automatically configure networking with Kubernetes

internally signed certifications. Select Advanced Network Configuration to customize the network interface types and TLS certificate generation

details. Complete the required values on S3, Management, and Replication tabs. NOTE: An object store will not support Bucket Replication features if ClusterIP is selected as the Service Type for

the Replication Receiver Service.

10. NOTE: If necessary, click Edit to modify any of the values for the object store that have been incorrectly set.

NOTE: Additionally, if you wish to save a portion of the object store settings as a template for future use, simply add a

name to the Save as template field at the bottom of this page of the wizard.

Figure 11. Creating a template

Finally, use the Review page to review the values to be used for configuring the new object store and click Save.

68 Working with object stores

The object store creation process begins and will take approximately 15 to 20 minutes to become available.

11. Optional: Refresh the UI to observe the various states of Health for the object store during the creation process such as Initializing, Starting, and Provisioning. The process is complete when the Health of the object store becomes Avaliable.

During object store creation, new pods for the object store become visible in the selected namespace.

Results

The new object store has been created. Before you can create buckets or use this object store, you must associate this object store with an IAM account. See Associate an IAM Account with an object store for more information.

Associate an IAM Account with an object store This task describes how to associate an IAM account with an object store in ObjectScale. After adding an account to an object store that account becomes a tenant within that object store. A tenant is a logical construct resulting from the binding of an account to an object store.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Select the namespace from the namespace drop-down on the top right corner of the ObjectScale UI.

4. Select the object store to add the IAM account to by clicking the object store name.

5. Select the Accounts tab. The Accounts list appears displaying the accounts currently associated with the object store.

6. Click Add. The Add Account to Object Store: wizard is displayed.

Working with object stores 69

Figure 12. Add an account to an object store

7. Complete the Add Account to Object Store wizard and click Save.

This process consists of: Select the Account ID, which will be the tenant to this account to be added to the object store. Type an Alias for this account. Enable/disable Encryption, as desired. By default the encryption status will show the account's encryption status. Default Bucket Quota limit for the account in the object store. Set the Notification at Quota. This is the quota at which a notification should be sent out. This can be set by providing

a quota value in the input box or as percentage of block writes at quota by selecting appropriate % from the drop-down. Set the Block writes at Quota limits for which writes must be blocked.

The selected IAM account is now associated with the object store and is a tenant of the object store.

View object store summary

About this task

The object store Summary page displays an at-a-glance view of the details about the configuration of the selected object store.

70 Working with object stores

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Select the namespace from the namespace drop-down on the top right corner of the ObjectScale UI.

4. Click the name of the object store that you want to review.

The Object Store Summary page is displayed. Here you can review high-level details about the select object store.

View S3 and management certificate properties You can view and configure the properties for S3 and management certificates.

About this task

To view the properties of an object store certificate:

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Select the namespace from the namespace drop-down on the upper right corner of the ObjectScale UI.

4. Click the name of the object store containing the certificates whose properties to view or to download.

5. In the right-most pane of the object store, click Certificates tab.

The Certificates tab consists of a S3, Management, and Replication Reciever sections.

Each of the sections of the Certificates tab shows the: a. Expiration details of the CA Bundle for S3/Management and Replication services. b. Certificate signing status displayed for each service.

If Kubernetes signed certificate is being used, which is the default configuration for object stores, S3 and Management sections allow you to Download CA Bundle. The certificate can be used for establishing trusted https connections for S3 and management service.

Edit an object store

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Select the namespace from the namespace drop-down on the top right corner of the ObjectScale UI.

4. Select the object store to edit by clicking the checkbox to the left of the object store name.

5. Click Edit.

The Edit Object Store wizard appears.

6. Go to the section in the Edit Object Store wizard containing the value of the object store to modify.

Working with object stores 71

Figure 13. Edit an object store

From this wizard, you can:

Click General to edit the general settings of the object store. The only value that can be edited here is the Group value. Click Storage to expand the object store, either horizontally or vertically.

NOTE: See Horizontally expand an object store capacity and Vertically expand an object store capacity for more

detailed instructions on expanding an object store.

Click Connectivity to modify the object store connectivity values.

7. Once complete, click Save to save the changes to the object store.

Set capacity alerts for an object store Capacity alerts are triggered once the object store consumption reaches the selected percentage of available capacity allowing you to make the necessary modifications to the object store contents or sizing before total capacity is reached.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Select the namespace from the namespace drop-down on the top right corner of the ObjectScale UI.

4. Click on the name of the object store.

72 Working with object stores

5. Click Capacity Alerts.

The Capacity Alerts page appears.

6. Enable or disable the Critical Alert setting. If you enable critical alerts, you must set the Critical Threshold percentage at which to send the alert.

A critical alert is triggered once the object store consumption reaches the selected percentage of available capacity.

7. Enable or disable the Warning Alert setting. If you enable warning alerts, you must set the Warning Threshold percentage at which to send the alert.

A warning alert is triggered once the object store consumption reaches the selected percentage of available capacity.

8. Once complete, click Save to save the changes to the object store capacity alert settings.

Or, click RESET undo any unsaved changes to the object store capacity alerts.

Delete an object store

About this task

Deleting the object store automatically deletes the storage (persistent volume claims [PVCs]) that are associated with it. The volumes and file systems are removed from the back-end storage as a result. Deleting an object store will not invoke garbage collection.

CAUTION: Deleting an object store deletes all associated user data, and the delete action is irreversible.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Select the namespace from the namespace drop-down on the top right corner of the ObjectScale UI.

4. Select the object store to delete by clicking the checkbox to the left of the object store name.

5. Click Delete and confirm in the dialog box that appears.

The object store disappears from the table before all the resources associated with it (pods, PVCs, volumes, so on) have been deleted. The removal operations continue in the background for a few minutes (depends on the size of the store). Removal can be monitored with kubectl (for example, using get pods).

Working with object stores 73

Working with Buckets in ObjectScale This chapter contains:

Topics:

About ObjectScale buckets Creating and managing buckets using ObjectScale About bucket policies Setting up bucket event notifications

About ObjectScale buckets Buckets are object containers that are used to control access to objects.

In S3, object containers are called buckets and this term has been adopted as a general term in ObjectScale. In ObjectScale, buckets are limited to S3 only.

A bucket is associated to only ObjectScale instances, the object store and account or tenant. An IAM user can create buckets only in the namespace to which the IAM user is assigned.

Bucket and object naming conventions

This topic details the rules that apply to the naming of ObjectScale buckets and their objects.

Bucket names

The following rules apply to the naming of buckets in ObjectScale:

Bucket names are required to be between three and 63 characters in length. Bucket names can include dots (.), hyphens (-), lowercase letters, and number characters ([a-z, 0-9]). Bucket names must begin and end with a number or lowercase letter characters ([a-z, 0-9]). Bucket names cannot be formatted as an IP address.

Object names

The following rules apply to the naming of ObjectScale S3 objects:

Cannot be null or an empty string Length range is 1..255 (Unicode character) No validation on characters.

Creating and managing buckets using ObjectScale

Create a bucket

Use the New Bucket wizard to set up new buckets.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

5

74 Working with Buckets in ObjectScale

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Select the namespace from the namespace drop-down on the top right corner of the ObjectScale UI.

4. Click the name of the object store that you want to contain the new bucket.

5. If the bucket account owner is not already added to the object store, click Accounts > Add.

Complete the Add Account to Object Store wizard that appears.

6. Otherwise, select the account from the Select an account dropdown in the Buckets tab.

To select an account other than the one currently displayed, first remove the selected account to see all accounts. Then, select the desired account either by using the dropdown, which displays all accounts, or begin typing the account id into to the Select an account field to dynamically filter the list of accounts.

7. In the object store details panel, from the right-most panel that is displayed in the UI, click the Buckets tab.

8. Click New Bucket. The New Bucket wizard appears.

Figure 14. Create a new bucket

9. In the General page, complete the required fields and then click Next:

Option Description

Name Type a name for the new bucket. Bucket names can consist only of lowercase letters, numbers, dots (.), and hyphens (-).

Object Store Name

Select the object store from the ObjectStore dropdown menu within which to create the bucket.

Bucket Owner Account

Select the bucket owner account from the Bucket Owner Account dropdown menu.

To select any account first remove selected account to see all accounts and click on the dropdown to list all accounts. Account can be selected from the data list or you can begin typing the account id to the Bucket Owner Account data list.

10. In the Policy page, describe the policy to apply to the bucket and then click Next.

Working with Buckets in ObjectScale 75

For more detailed information on creating a bucket policy statement, see About bucket policies.

a. Switch to the Text view of the policy editor by toggling between the View and Text views of the policy editor. b. In the Policy editor text field, type the JSON-formatted policy or copy and paste a previously created policy. The syntax

used for policies is the same as that used for Amazon AWS. c. Provided your policy is valid, you can switch to the tree view of the Policy. The tree view makes it easier to view your

policy and to expand and contract statements.

11. In the Controls page, complete the required fields and then click Next.

Option Description

Versioning Enable versioning to maintain multiple versions of the same object within the bucket.

Versioning is disabled by default.

NOTE: Versioning cannot be Off when Object Lock is On

Object Lock Enable object lock which allows objects to be locked or protected from deletion or overwrite, for a fixed amount of time or indefinitely, depending on the configuration.

Set the object lock retention mode for the objects within this bucket, to:

GOVERNANCE COMPLIANCE NONE (object lock enabled, but the retention mode will not be set)

Set the retention period, select either Days or Years and type the number of days/years for the period, if GOVERNANACE or COMPLIANCE is selected.

Object Lock is disabled by default.

Quotas Enable quotas for the bucket. When enabled, you can set the storage level or object count level in ObjectScale which will Block writes at Quota, a hard quota, and Notification at Quota, a soft quota.

Quotas is disabled by default.

If Default Bucket Quota is set at the account level while adding it to object store, same value will be set for Block Writes At and you can update quota values of disable quota in the bucket.

Encryption Enable encryption to save the bucket data in an encrypted format. If Encryption was enabled at the account level when add to object store, you cannot disable encryption in the bucket.

Encryption is disabled by default.

NOTE: Encryption cannot be disabled after you create the bucket with encryption enabled.

12. In the Event Rule page, complete the required fields and then click Next.

For more detailed information on editing a bucket event notifications, see Setting up bucket event notifications.

You must have at least one notification destination for this account to set up a bucket event rule.

Option Description

Event Rule Name Type a name for the new event.

Events Select one or more event types that will trigger this event notification.

Prefix/Suffix Type the object prefix or suffix values that will trigger this event notification.

Send To Select the notification destination to be used to send the notifications for the configured event(s).

NOTE: Click on Add Event Rule to add multiple event rules. But, you cannot create configurations on the same bucket

that share a common event type.

13. Finally, use the Review page to review the values to be used for configuring the new bucket and click Save.

If necessary, click Edit to modify any of the values for the bucket that have been incorrectly set.

Results

The new bucket is created in the object store and can now be viewed in the object store Buckets tab.

76 Working with Buckets in ObjectScale

View the Bucket Summary

You can view the details for an existing bucket.

About this task

To view the properties of an object store bucket:

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Select the namespace from the namespace drop-down on the top right corner of the ObjectScale UI.

4. Click the name of the object store containing the bucket whose properties to view.

5. In the right-most pane for the object store click Buckets tab.

6. Select the account from the Accounts dropdown in the Buckets tab.

To select an account other than the one currently displayed, first remove the selected account to see all accounts. Then, select the desired account either by using the dropdown, which displays all accounts, or begin typing the account id into to the Select an account field to dynamically filter the list of accounts.

7. Click on the name of the bucket to view the bucket Summary.

Edit a bucket

Use the Edit Bucket wizard to edit details of existing buckets.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Select the namespace from the namespace drop-down on the top right corner of the ObjectScale UI.

4. Click the name of the object store that contains the bucket to be modified.

5. In the right-most pane for the object store click Buckets tab.

6. Select the account from the Accounts dropdown in the Buckets tab.

To select an account other than the one currently displayed, first remove the selected account to see all accounts. Then, select the desired account either by using the dropdown, which displays all accounts, or begin typing the account id into to the Select an account field to dynamically filter the list of accounts.

7. Select the bucket to modify and click Edit. The Edit Bucket wizard appears.

8. Select one or more of the sections with the bucket values to modify:

General (review only, no edit) Policy Controls Event Rules

9. In the Policy page, modify the policy to apply to the bucket and then click Next.

For more detailed information on editing a bucket policy statement, see About bucket policies.

a. Switch to the Text view of the policy editor by toggling between the View and Text views of the policy editor. b. In the Policy editor text field, type the JSON-formatted policy or copy and paste a previously created policy. The syntax

used for policies is the same as that used for Amazon AWS.

Working with Buckets in ObjectScale 77

c. Provided your policy is valid, you can switch to the tree view of the Policy. The tree view makes it easier to view your policy and to expand and contract statements.

10. In the Controls page, modify the required fields and then click Next.

Option Description

Versioning Enable versioning to maintain multiple versions of the same object within the bucket.

Versioning is disabled by default.

NOTE: Versioning cannot be Off when Object Lock is On

Object Lock Enable object lock which allows objects to be locked or protected from deletion or overwrite, for a fixed amount of time or indefinitely, depending on the configuration.

Set the object lock retention mode for the objects within this bucket, to:

GOVERNANCE COMPLIANCE NONE (object lock enabled, but the retention mode will not be set)

Set the retention period, select either Days or Years and type the number of days/years for the period, if GOVERNANACE or COMPLIANCE is selected.

If object lock is enabled during bucket creation then only retention period and object lock modes can be modified and you cannot disable object lock once enabled.

Quotas Enable quotas for the bucket. When enabled, you can set the storage level or object count level when ObjectScale will Block writes at Quota, a hard quota, and Notification at Quota, a soft quota.

11. In the Event Rule page, modify the required fields and then click Next.

For more detailed information on editing a bucket event notifications, see Setting up bucket event notifications.

You must have at least one notification destination for this account to set up a bucket event rule.

Option Description

Event Rule Name Type a name for the new event.

Events Select one or more event types that will trigger this event notification.

Prefix/Suffix Type the object prefix or suffix values that will trigger this event notification.

Send To Select the notification destination to be used to send the notifications for the configured event(s).

12. In the Review page, expand each of the available fields and review your values for the new bucket.

If you need to modify any of the bucket values, click Edit to return to that portion of the New Bucket wizard. After changing the value(s), return to the Review page.

13. After making the necessary changes, click Save.

Delete a bucket

Delete a bucket when the object store no longer needs the bucket.

Prerequisites

The bucket must be empty. ObjectScale will only allow the deletion of buckets without any data within them.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Select the namespace from the namespace drop-down on the top right corner of the ObjectScale UI.

78 Working with Buckets in ObjectScale

4. Click the name of the object store that contains the bucket to be deleted.

5. In the right-most pane for the object store click Buckets tab.

6. Select the account from the Accounts dropdown in the Buckets tab.

To select an account other than the one currently displayed, first remove the selected account to see all accounts. Then, select the desired account either by using the dropdown, which displays all accounts, or begin typing the account id into to the Select an account field to dynamically filter the list of accounts.

7. Select the bucket to be deleted and click Delete. ObjectScale prompts Are you sure you want to delete following bucket(s)?

8. In the Delete Bucket confirmation window, confirm that the appropriate bucket will be deleted.

Results

The bucket is deleted from the object store.

About bucket policies The ObjectScale provides a bucket policy editor to enable you to create a bucket policy for a bucket, either new or existing.

Bucket policies provide fine grained control over permissions for bucket operations and for operations on objects within the bucket. Policy conditions are used to assign permissions for a range of objects that match the condition and are used to automatically assign permissions to newly uploaded objects.

Policies are defined in JSON format in the Text view of the policy editor. Once defined a policy can be viewed in the Tree view. The syntax used for policies is the same as that used for Amazon AWS. The operations for which permissions can be assigned are limited to those operations supported by ObjectScale.

The bucket policy editor has a code view and a tree view.

The code view enables you to enter JSON policies from scratch or to paste existing policies into the editor and modified. For example, if you have existing policies in JSON format, you can paste them into the code view and modify them.

The tree view provides a mechanism for navigating a policy and is useful where you have a large number of statements in a policy. You can expand and contract the statements and search them.

Create or edit a bucket policy

You can create or modify a bucket policy for an existing bucket using the Policy editor.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Select the namespace from the namespace drop-down on the top right corner of the ObjectScale UI.

4. Click the name of the object store that contains the bucket to be modified.

5. In the right-most pane for the object store click Buckets tab.

6. Select the account from the Accounts dropdown in the Buckets tab.

To select an account other than the one currently displayed, first remove the selected account to see all accounts. Then, select the desired account either by using the dropdown, which displays all accounts, or begin typing the account id into to the Select an account field to dynamically filter the list of accounts.

7. Select the bucket to modify and click Edit. The Edit Bucket wizard appears.

8. In the Policy page, modify the policy to apply to the bucket and then click Next.

a. Switch to the Text view of the policy editor by toggling between the View and Text views of the policy editor. b. In the Policy editor text field, type the JSON-formatted policy or copy and paste a previously created policy. The syntax

used for policies is the same as that used for Amazon AWS.

Working with Buckets in ObjectScale 79

c. Provided your policy is valid, you can switch to the tree view of the Policy. The tree view makes it easier to view your policy and to expand and contract statements.

9. Save.

Bucket policy support

ObjectScale supports the setting of S3 bucket access policies. Unlike ACLs, which either permit all actions or none, access policies provides specific users, or all users, conditional and granular permissions for specific actions. Policy conditions can be used to assign permissions for a range of objects that match the condition and can be used to automatically assign permissions to newly uploaded objects.

How access to resources is managed when using the S3 protocol is described in https://docs.aws.amazon.com/AmazonS3/ latest/dev/s3-access-control.html and you can use the information as the basis for understanding and using S3 bucket policies. This section provides basic information about the use of bucket policies, and to identify the differences when using bucket policies.

The following provides an example of a bucket policy:

{ "Version": "2012-10-17", "Id": "S3PolicyIdNew2", "Statement":[ { "Sid":"Granting PutObject permission to user2 ", "Effect":"Allow", "Principal": "user_n2", "Action":["s3:PutObject"], "Resource":["PolicyBuck1/*"], "Condition": { "StringEquals": {"s3:x-amz-server-side-encryption": [ "AES256"]} } } ] }

Each policy is a JavaScript Object Notation (JSON) document that includes a version, an identifier, and one or more statements.

Version The Version field specifies the policy language version and can be either 2012-10-17 or 2008-10-17. If the version is not specified, 2008-10-17 is automatically inserted.

It is good practice to set the policy language for a new policy to the latest version, 2012-10-17.

Id The Id field is optional.

Each Statement includes the following elements:

SID A statement ID is a string that describes what the statement does.

Resources The bucket or object that is the subject of the statement. The resource can be associated with a Resource or NotResource statement.

The resource name is the bucket and key name and is specified differently depending on whether you are using virtual host style addressing or path style addressing, as shown:

Host Style: http://bucketname.ns1.emc.com/objectname Path Style: http://ns1.emc.com/bucketname/objectname

In either case, the resource name is: bucketname/objectname.

You can use the (*) and (?) wildcard characters, where asterisk (*) represents any combination of zero or more characters and a question mark (?) represents any single character. For example, you can represent all objects in bucket that is called bucket name, using:

bucketname/*

Actions The set of operations that you want to assign permissions to (enable or deny). The supported operations are listed in Supported bucket policy operations.

80 Working with Buckets in ObjectScale

The operation can be associated with an Action or NotAction statement.

Effect Can be set to Allow or Deny to determine whether you want to enable or deny the specified actions.

Principal The user who is enabled or denied the specified actions.

To grant permissions to everyone, as anonymous access, you can set the principal value to a wildcard, "*", as shown:

"Principal":"*"

Conditions The condition under which the policy is in effect. The condition expression is used to match a condition that is provided in the policy with a condition that is provided in the request.

The following condition operators are not supported: Binary, ARN, IfExists, Check Key Exists. The supported condition keys are listed in Supported bucket policy conditions.

NOTE: ObjectScale bucket policies do not support federated users, nor do they support Amazon IAM users and roles.

More information about the elements that you can use in a policy are described in the Amazon S3 documentation, https:// docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements.html.

Bucket policy scenarios

In general, the bucket owner has full control on a bucket and can grant permissions to other users and can set S3 bucket policies using an S3 client. It is also possible to set bucket policies using the bucket policy editor in the New Bucket and Edit Bucket wizards in the ObjectScale UI.

You can use bucket policies in the following typical scenarios:

Grant bucket permissions to a user Grant bucket permissions to all users Automatically assign permissions to created objects

Grant bucket permissions to a user

To grant permission on a bucket to a user apart from the bucket owner, specify the resource that you want to change the permissions for. Set the principal attribute to the name of the user, and specify one or more actions that you want to enable.

The following example shows a policy that grants a user who is named user1 the permission to update and read objects in the bucket that is named mybucket:

{ "Version": "2012-10-17", "Id": "S3PolicyId1", "Statement": [ { "Sid": "Grant permission to user1", "Effect": "Allow", "Principal": ["user1"], "Action": [ "s3:PutObject","s3:GetObject" ], "Resource":[ "mybucket/*" ] } ] }

You can also add conditions. For example, if you only want the user to read and write object when accessing the bucket from a specific IP address, add a IpAddress condition as shown in the following policy:

{ "Version": "2012-10-17", "Id": "S3PolicyId1", "Statement": [ { "Sid": "Grant permission ", "Effect": "Allow",

Working with Buckets in ObjectScale 81

"Principal": ["user1"], "Action": [ "s3:PutObject","s3:GetObject" ], "Resource":[ "mybucket/*" ] "Condition": {"IpAddress": {"aws:SourceIp": " "} } ] }

Grant bucket permissions to all users

To grant permission on a bucket to a user apart from the bucket owner, specify the resource that you want to change the permissions for. Set the principal attribute as anybody (*), and specify one or more actions that you want to enable.

The following example shows a policy that grants anyone permission to read objects in the bucket that is named mybucket:

{ "Version": "2012-10-17", "Id": "S3PolicyId2", "Statement": [ { "Sid": "statement2", "Effect": "Allow", "Principal": ["*"], "Action": [ "s3:GetObject" ], "Resource":[ "mybucket/*" ] } ] }

Automatically assign permissions to created objects

You can use bucket policies to automatically enable access to ingested object data. In the following example bucket policy, user1 and user2 can create subresources (that is, objects) in the bucket that is named mybucket and can set object ACLs. With the ability to set ACLs, the users can then set permissions for other users. If you set the ACL in the same operation, a condition can be set. Such that a canned ACL public-read must be specified when the object is created. This ensures anybody can read all the created objects.

{ "Version": "2012-10-17", "Id": "S3PolicyId3", "Statement": [ { "Sid": "statement3", "Effect": "Allow", "Principal": ["user1", "user2"], "Action": [ "s3:PutObject, s3:PutObjectAcl" ], "Resource":[ "mybucket/*" ] "Condition":{"StringEquals":{"s3:x-amz-acl":["public-read"]}} } ] }

Supported bucket policy operations

Table 20. Permissions for Object Operations

Permission keyword Supported S3 operations

s3:GetObject applies to latest version for a version-enabled bucket

GET Object, HEAD Object

s3:GetObjectVersion GET Object, HEAD Object This permission supports requests that specify a version number

82 Working with Buckets in ObjectScale

Table 20. Permissions for Object Operations (continued)

Permission keyword Supported S3 operations

s3:PutObject PUT Object, POST Object, Initiate Multipart Upload, Upload Part, Complete Multipart Upload PUT Object - Copy

s3:GetObjectAcl GET Object ACL

s3:GetObjectVersionAcl GET ACL (for a Specific Version of the Object)

s3:PutObjectAcl PUT Object ACL

s3:PutObjectVersionAcl PUT Object (for a Specific Version of the Object)

s3:DeleteObject DELETE Object

s3:DeleteObjectVersion DELETE Object (a Specific Version of the Object)

s3:ListMultipartUploadParts List Parts

s3:AbortMultipartUpload Abort Multipart Upload

Table 21. Permissions for Bucket Operations

Permission keyword Supported S3 operations

s3:DeleteBucket DELETE Bucket

s3:ListBucket GET Bucket (List Objects), HEAD Bucket

s3:ListBucketVersions GET Bucket Object versions

s3:GetLifecycleConfiguration GET Bucket lifecycle

s3:PutLifecycleConfiguration PUT Bucket lifecycle

Table 22. Permissions for Bucket Sub-resource Operations

Permission keyword Supported S3 operations

s3:GetBucketAcl GET Bucket acl

s3:PutBucketAcl PUT Bucket acl

s3:GetBucketCORS GET Bucket cors

s3:PutBucketCORS PUT Bucket cors

s3:GetBucketVersioning GET Bucket versioning

s3:PutBucketVersioning PUT Bucket versioning

s3:GetBucketPolicy GET Bucket policy

s3:DeleteBucketPolicy DELETE Bucket policy

s3:PutBucketPolicy PUT Bucket policy

Supported bucket policy conditions

The condition element is used to specify conditions that determine when a policy is in effect.

The following tables show the condition keys that are supported and that can be used in condition expressions.

Table 23. Supported generic AWS condition keys

Key name Description Applicable operators

aws:CurrentTime Used to check for date/time conditions Date operator

aws:EpochTime Used to check for date/time conditions using a date in epoch or UNIX time (see Date Condition Operators).

Date operator

Working with Buckets in ObjectScale 83

Table 23. Supported generic AWS condition keys (continued)

Key name Description Applicable operators

aws:principalType Used to check the type of principal (user, account, federated user, etc.) for the current request.

String operator

aws:SourceIp Used to check the requester's IP address. String operator

aws:UserAgent Used to check the requester's client application. String operator

aws:username Used to check the requester's user name. String operator

Table 24. Supported S3-specific condition keys for object operations

Key name Description Applicable permissions

s3:x-amz-acl Sets a condition to require specific access permissions when the user uploads an object.

s3:PutObject, s3:PutObjectAcl, s3:PutObjectVersionAcl

s3:x-amz-grant-permission (for explicit permissions), where permission can be:read, write, read-acp, write-acp, full- control

Bucket owner can add conditions using these keys to require certain permissions.

s3:PutObject, s3:PutObjectAcl, s3:PutObjectVersionAcl

s3:x-amz-server-side-encryption Requires the user to specify this header in the request.

s3:PutObject, s3:PutObjectAcl

s3:VersionId Restrict the user to accessing data only for a specific version of the object

s3:PutObject, s3:PutObjectAcl, s3:DeleteObjectVersion

Table 25. Supported S3-specific condition keys for bucket operations

Key name Description Applicable permissions

s3:x-amz-acl Set a condition to require specific access permissions when the user uploads an object

s3:CreateBucket, s3:PutBucketAcl

s3:x-amz-grant-permission (for explicit permissions), where permission can be:read, write, read-acp, write-acp, full- control

Bucket owner can add conditions using these keys to require certain permissions

s3:CreateBucket, s3:PutBucketAcl

s3:prefix Retrieve only the object keys with a specific prefix.

s3:ListBucket, s3:ListBucketVersions

s3:delimiter Require the user to specify the delimiter parameter in the Get Bucket (List Objects) request.

s3:ListBucket, s3:ListBucketVersions

s3:max-keys Limit the number of keys returned in response to the Get Bucket (List Objects) request by requiring the user to specify the max-keys parameter.

s3:ListBucket, s3:ListBucketVersions

Setting up bucket event notifications ObjectScale supports the configuration of bucket level event notification to allow you to easily monitor when certain configurable events occur within the bucket, such as when objects are created or deleted within the bucket. Bucket event notifications can be utilized to build out distributed and decoupled modern applications.

To set up ObjectScale's bucket event notification feature, you must configure two independent components:

Configure the destination WebHook server Configure the Bucket Event Notifications

84 Working with Buckets in ObjectScale

Configure the destination WebHook server Overview

For ObjectScale the only supported destination is a WebHook. WebHooks are a way to receive information when it happens, rather than continually polling for that data.

The following is the expected schema for configuring the WebhookConfig element:

token 1000 comment http://10.55.66.77:3000/hook MyWebhook

WebHook Config syntax Description

Name Identifier which uniquely identifies this destination among all other WebHook configured within an account in the object store.

Endpoint Webhook server endpoint.

AuthToken Opaque string or JWT authorization token.

BackupLimit Maximum limit size for undelivered messages.

Comment (Optional) Comment to this setting.

The WebHook name must be unique among WebHooks within an account in the object store. Once configured, ObjectScale will internally build the element, which you can collect with a GET.

ObjectScale supports these Destination Configuration Manager (DCM) APIs for interacting with the destination WebHook server:

GetEventDestinationConfiguration (type, name) DeleteEventDestinationConfiguration (type, name) ListEventDestinationConfigurations (type) PutEventDestinationConfiguration (type, name, config)

Configure the Bucket Event Notifications Overview

You can use the ObjectScale UI or the S3 API for PutBucketNotificationConfiguration and/or GetBucketNotificationConfiguration to configure bucket event notifications by creating an Event Rule that will trigger an event notification when the event occurs.

Each bucket uses an Event Rule section with the required fields to configure event notifications for that bucket. Using the ObjectScale UI, you can create a new bucket or modify an existing bucket to add, edit, or delete event rules.

Working with Buckets in ObjectScale 85

Figure 15. Edit Bucket - Event Rule

Event Rule Name - Type a name for the new event. Events - Select one or more event types that will trigger this event notification. Prefix/Suffix - Type the object name prefix or suffix values that will trigger this event notification. Send To - Select the notification destination to be used to send the notifications for the configured event(s).

You must have at least one notification destination for this account to set up a bucket event notification rule.

NOTE: Click on Add Event Rule to add multiple event rules. But, you cannot create configurations on the same bucket

that share a common event type.

When using the API, a notification configuration for a bucket is an XML document describing zero or more topic configurations. If no configuration is set on a bucket, it will implicitly have an empty NotificationConfiguration element.

For example, shown below is the request.body of the NotificationConfiguration without any topic configurations:

When creating new bucket notifications, use the following syntax for the request.body of the BucketNotificationConfiguration xml:

Name

86 Working with Buckets in ObjectScale

event-type ... (prefix|suffix) string ... webhook-urn ...

NotificationConfiguration syntax Description

TopicConfiguration May contain zero or more TopicConfigurations

ID Optional. If unspecified, ObjectScale will generate one

Event Must be one or more from s3:ObjectCreated:* s3:ObjectCreated:Put s3:ObjectCreated:Copy s3:ObjectCreated:CompleteMultipartUpload s3:ObjectRemoved:* s3:ObjectRemoved:Delete s3:ObjectRemoved:DeleteMarkerCreated s3:Replication:OperationFailedReplication

Topic Webhook URN referring to a webhook configuration in DCM

Filter Optional. May contain 0 or 1 S3Key filters

S3Key Optional. May contain 0-2 FilterRules

FilterRule Optional.

Name Optional. Must be one of prefix | suffix. Only one FilterRule of each type may be specified in an S3Key filter.

Value Optional.

For example, the following NotificationConfiguration file shows how to configure a notification to a webhook any time an object is created.

CreateEvents createWebhook s3:ObjectCreated:*

About bucket event notifications

At the highest level, an event is a change to the state of an object within a bucket.

Event Types

ObjectScale provides event notifications for the following types of events.

Working with Buckets in ObjectScale 87

Supported Event Type Description

s3:ObjectCreated:Put An object is created via an S3 PUT operation

s3:ObjectCreated:Copy An object is created via an S3 COPY operation

s3:ObjectCreated:CompleteMultipartU pload

An object is created via an S3 CompleteMultipartUpload operation

s3:ObjectCreated:* Any time an object is created

s3:ObjectRemoved:Deleted Any time a non-versioned object is deleted or an object version is permanently deleted

s3:ObjectRemoved:DeleteMarkerCreat ed

Any time a delete marker is created for a versioned object

s3:ObjectRemoved:* Any time an object is deleted

s3:Replication:ObjectFailedReplication Any time an object fails replication

Event Notification Structure

ObjectScale's event notification structure conforms to the S3 event notification structure standard.

Below shows an example notification structure:

{ "Records": [ { "eventVersion": "2.2", "eventSource": "aws:s3", "awsRegion": "us-west-2", "eventTime": "2021-02-12T02:14:48.398Z", "eventName": "s3:ObjectCreated:Put", "userIdentity": { "principalId": "urn:ecs:iam::ad126a31-0286-4567-9670-c6032d1d89ac:root" }, "requestParameters": { "sourceIPAddress": "172.17.0.1" }, "responseElements": { "x-amz-request-id": "ac11001b:17793e42a6a:a7:147", "x-amz-id-2": "87fec1b544f39058bab52f8dec0a0e257a3703454c40e260355f1578bc597406" }, "s3": { "s3SchemaVersion": "1.0", "configurationId": "MyConfiguration1", "bucket": { "name": "bucket01", "ownerIdentity": { "principalId": "urn:ecs:iam::ad126a31-0286-4567-9670-c6032d1d89ac:root" }, "arn": ".bucket01" }, "object": { "key": "object-for-notification", "size": "10", "etag": "c239368c6b3ec9b9dbc5a6b799e3756a", "versionId": "AAABd5QE804oTME0iiFB2rY0z1_bH-nEK7w", "sequencer": "100000000000000000000000000000000000000000072e338" } } } ] }

88 Working with Buckets in ObjectScale

Configure Webhook Destination for S3 Notifications

Set up the destination configuration for the WebHooks server that will receive the bucket event notifications.

Prerequisites

You must have configured:

an IAM user with an Access Key and Secret. the destination configuration of the webhooks server. The URN for the destination configuration is a required field in the

bucket notification configuration.

About this task

Using a Linux workstation with kubectl and s3curl.pl, and that has access to ObjectScale on this k8s cluster:

Steps

1. List the objectscale-gateway service endpoint.

kubectl -n get svc objectscale-gateway

2. Set an environment variable for the DCM endpoint.

DCM_ENDPOINT=

3. Set your user Access Key and Secret that you created during the s3curl setup.

ACCESS_KEY=

SECRET=

4. Calculate the signature for the request to add a new WebHook destination configuration.

NL=$'\n' RESOURCE=/destconf/webhook/ DATEVAL="`date -u ${adj} +'%a, %d %b %Y %H:%M:%S %z'`" STRING_TO_SIGN="PUT${NL}${NL}application/xml;charset=utf-8${NL}${DATEVAL}${NL}$ {RESOURCE}" SIGNATURE=`/bin/echo -n "$STRING_TO_SIGN" | openssl sha1 -hmac ${SECRET} -binary | base64`

5. Add a new WebHook destination configuration to DCM passing the WebhookConfig XML in the request payload.

CREATE_WEBHOOK_OUTPUT=$(curl -v -H "Date: ${DATEVAL}" -H "Authorization: AWS ${ACCESS_KEY}:${SIGNATURE}" -H "Content-Type: application/xml;charset=utf-8" -d @ -X PUT https://${DCM_ENDPOINT}: / destconf/webhook/)

6. Review the reply from the request.

The reply should look like:

urn:objectscale:webhook::24069e07-2b7a-4dc4-98ef-ee7d4017cf96:MyWebhook2 token 100000 optional comment http://10.247.102.238:3000/hook MyWebhook2

Working with Buckets in ObjectScale 89

7. Save the Urn of the WebHook configuration created in 5.

When creating the bucket notification configuration(s) in Create a bucket notification configuration using the ObjectScale APIs , use the value from $WEBHOOK_URN in the of the desired TopicConfiguration.

WEBHOOK_URN=$(echo $CREATE_WEBHOOK_OUTPUT | xmllint -format - | grep Urn | sed 's/ \(.*\)<\/Urn>/\1/g' | sed -e 's/^[ \t]*//')

8. Optional: Review the WebHook destination configuration:

Use the ${DCM_ENDPOINT} value from 2.

s3curl.pl --ord --debug --id=${ACCESS_KEY} --key=${SECRET} -- https://$ {DCM_ENDPOINT}: /destconf/webhook/${WEBHOOK_URN}

9. Optional: If/when you need to remove a WebHook destination configuration, delete a webhook configuration by:

Use the ${DCM_ENDPOINT} value from 2.

s3curl.pl --ord --debug --id=${ACCESS_KEY} --key=${SECRET} --delete -- https://$ {DCM_ENDPOINT}: /destconf/webhook/${WEBHOOK_URN}

Create a bucket notification configuration using the ObjectScale APIs

Prerequisites

Before you can create bucket notification configuration, you must have:

Created an object store and a bucket within the object store. Set up the WebHook destination configuration.

Steps

1. Create the NotificationConfigurations file with your config xml.

When creating the bucket notification configuration(s), use the value from $WEBHOOK_URN in the of the desired TopicConfiguration.

ObjectScale provides event notifications for the following types of events.

Supported Event Type Description

s3:ObjectCreated:Put An object is created via an S3 PUT operation

s3:ObjectCreated:Copy An object is created via an S3 COPY operation

s3:ObjectCreated:CompleteMultipart Upload

An object is created via an S3 CompleteMultipartUpload operation

s3:ObjectCreated:* Any time an object is created

s3:ObjectRemoved:Deleted Any time a non-versioned object is deleted or an object version is permanently deleted

s3:ObjectRemoved:DeleteMarkerCrea ted

Any time a delete marker is created for a versioned object

s3:ObjectRemoved:* Any time an object is deleted

The example below shows a configuration which listens for all ObjectCreated and ObjectRemoved events.

90 Working with Buckets in ObjectScale

MyConfiguration s3:ObjectCreated:* s3:ObjectRemoved:* urn:objectscale:webhook::722d25f2-9c5b-41fe-82ac-605782945488:MyWebhook

2. Put the notification configuration to the bucket

s3curl.pl --ord --debug --id=${ACCESS_KEY} --key=${SECRET} --calculateContentMd5 -- put= -- http://$(kubectl get svc |awk '/-s3/{print $4}'):80/${BUCKET}? notification -v

3. Verify the bucket notification configuration was set in the bucket.

s3curl.pl --ord --debug --id=${ACCESS_KEY} --key=${SECRET} http://$(kubectl get svc | awk '/-s3/{print $4}'):80/{BUCKET}?notification -v | xmllint -format -

Results

You will now receive notification record output like this in your WebHook server as users perform S3 operations in the monitored bucket. For an example webhook server listening, see Sample setting of simple listener at Webhook server.

Received notification #7 from ::ffff:172.17.0.38 { "Records": [ { "eventVersion": "2.2", "eventSource": "aws:s3", "awsRegion": "us-west-2", "eventTime": "2021-02-12T02:14:48.398Z", "eventName": "s3:ObjectCreated:Put", "userIdentity": { "principalId": "urn:ecs:iam::ad126a31-0286-4567-9670-c6032d1d89ac:root" }, "requestParameters": { "sourceIPAddress": "172.17.0.1" }, "responseElements": { "x-amz-request-id": "ac11001b:17793e42a6a:a7:147", "x-amz-id-2": "87fec1b544f39058bab52f8dec0a0e257a3703454c40e260355f1578bc597406" }, "s3": { "s3SchemaVersion": "1.0", "configurationId": "MyConfiguration1", "bucket": { "name": "bucket01", "ownerIdentity": { "principalId": "urn:ecs:iam::ad126a31-0286-4567-9670-c6032d1d89ac:root" }, "arn": ".bucket01" }, "object": { "key": "object-for-notification", "size": "10", "etag": "c239368c6b3ec9b9dbc5a6b799e3756a", "versionId": "AAABd5QE804oTME0iiFB2rY0z1_bH-nEK7w", "sequencer": "100000000000000000000000000000000000000000072e338" } } } ] }

Working with Buckets in ObjectScale 91

Sample setting of simple listener at Webhook server

NOTE: Requires npm and nodejs installed:

mkdir webhook cd webhook npm init -y npm install express body-parser # create index.js from the linked file node index.js Webhook listening on :3000 /hook

Content of index.js

const express = require("express") const bodyParser = require("body-parser")

const app = express() const PORT = 3000 const PATH = "/hook"

count = 0

app.use(bodyParser.json())

app.post(PATH, (req, res) => { count++ console.log("Received notification #" console.log(JSON.stringify(req.body, null, 2)) res.status(200).end() })

app.listen(PORT, () => console.log(`Webhook listening on :${PORT} ${PATH}`))

92 Working with Buckets in ObjectScale

Working with ObjectScale Systems This chapter contains details on creating a federation of multiple ObjectScale systems, in these topics:

Topics:

ObjectScale Systems Create a federation of ObjectScale systems Add additional ObjectScale instances to an existing ObjectScale federation

ObjectScale Systems When you create a federation of ObjectScale systems it allows IAM entities to replicate from one ObjectScale system to other ObjectScale systems in the federation.

The ObjectScale System page contains details on the federation status of an ObjectScale system. It also displays details on any other ObjectScale systems that are a part of the federation of ObjectScale systems. The ObjectScale Systems tab allows you to create and join federations.

Within a federation, an ObjectScale instance can either be the primary instance or a secondary instance. There can be two or more secondary instances within a federation, but there can only be a single primary instance. Any instance not in a federation appears as Not Federated, which is the default state. After you create a federation, a heartbeat will be maintained between trusted ObjectScale instances to track their connectivity status (Online or Offline).

Figure 16. ObjectScale Systems

6

Working with ObjectScale Systems 93

Federating ObjectScale systems relies on external endpoints provided and maintained by the ObjectScale Gateway service. This ObjectScale Gateway is made of two services:

objectscale-gateway-internal objectscale-gateway

ObjectScale Gateway and ObjectScale Gateway Internal services are separate paths for forwarding requests to ObjectScale- level services. The ObjectScale Gateway service will have either Kubernetes-signed certificate or an externally signed certificate. It is used as a customer-facing endpoint. The ObjectScale Gateway Internal service has internally signed certificate and is used for ObjectScale-to-ObjectScale communication.

Here is an overview of what an Administrator user must do to create a federation of ObjectScale systems:

NOTE: Access to both the primary and secondary ObjectScale systems is required.

1. From the ObjectScale system you want to make the primary system in the federation, create the federation. It will become the primary ObjectScale instance.

2. From the ObjectScale system you want to make a secondary system in the federation, join a federation. This downloads the FederationSigning_ .xml file that the primary ObjectScale system must sign.

3. From the primary ObjectScale system, upload FederationSigning_ .xml request to the primary ObjectScale. The primary ObjectScale generates a signed certificate that is named PostFederationSigning_ .xml and automatically download the file.

4. From the secondary ObjectScale system, add a new remote instance by uploading the signed certificate that was downloaded from the primary ObjectScale system. The ObjectScale type will now show Secondary.

5. Finally, from each system you must initiate trust of the remote instances from each ObjectScale system.

Once the Remote Instance on both sides have Trust Initiated Trust Status, the federation service on each side tries to communicate to the remote instance over the objectscale-gateway-internal endpoint. If this system-to-system communication is successful, it moves each remote instance to the Trusted Trust Status. Communication is successful once both ObjectScale instances can verify the certificates in the trusted list.

You can add additional ObjectScale instances to this federation by repeating this process.

Use the following tasks to create and maintain a federation of ObjectScale systems.

Create a federation of ObjectScale systems Creating a federation of ObjectScale systems allows you to replicate IAM entities from one ObjectScale system to other systems.

About this task

An Administrator user on both ObjectScale systems will need to complete these steps to create the federation of ObjectScale systems.

Steps

Complete these steps on the ObjectScale system you wish to make the primary ObjectScale instance in the federation.

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Systems. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select ObjectScale Systems. For ObjectScale Portal UI: Click the ObjectScale Systems tab. A user can see a list of ObjectScale Systems that the user is authorized to view.

3. Click Create Federation and click YES to create the necessary federation CA and establish this ObjectScale instance as the primary.

94 Working with ObjectScale Systems

Figure 17. Create Federation

After you click Create Federation this ObjectScale system becomes the primary instance for the federation. You will no longer be able create additional federation CAs or join this ObjectScale instance to any other federations.

4. Ensure the instance is now listed as Primary in the ObjectScale Systems page.

Complete these steps on the ObjectScale system you wish to make a secondary ObjectScale instance in the federation.

5. Log in to the ObjectScale Portal UI or VMware vSphere Client.

6. Go to the ObjectScale Systems. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select ObjectScale Systems. For ObjectScale Portal UI: Click the ObjectScale Systems tab. A user can see a list of ObjectScale Systems that the user is authorized to view.

7. Click Join Federation and download the federation signing request file, FederationSigning_ .xml.

Ensure that this file is accessible by both ObjectScale instances.

8. From the primary ObjectScale instance, click New Remote Instances and upload the federation signing request file from the other ObjectScale instance. After uploading the xml file, click SAVE.

Figure 18. New Remote Instance

Once you have uploaded this file, the primary ObjectScale instance automatically generates and downloads the signed request file, PostFederationSigning_ .xml. Additionally, the Secondary ObjectScale instance will now appear as a Not Trusted remote instance of ObjectScale within the federation managed by the primary ObjectScale instance.

9. From the secondary ObjectScale instance, click New Remote Instances and upload the signed federation signing request file from the primary ObjectScale instance. The primary ObjectScale instance will now appear as a Not Trusted remote instance of ObjectScale within the federation.

10. Finally, to complete the federation process linking these two ObjectScale instances, establish trust between the two ObjectScale instances.

Working with ObjectScale Systems 95

a. From the Secondary ObjectScale instance, select the checkbox of the primary ObjectScale instance that needs to establish trust with the Secondary ObjectScale instance and click Initiate Trust.

The Initiate Trust window appears.

b. Verify that the correct details are shown for the selected ObjectScale instance you wish to establish trust with in the federation and click Yes.

Figure 19. Initiate trust with the primary ObjectScale

c. From the primary ObjectScale instance, select the checkbox of the Secondary ObjectScale instance that needs to establish trust with the primary ObjectScale instance and click Initiate Trust.

d. Verify that the correct details are shown for the selected ObjectScale instance you wish to establish trust with in the federation and click Yes.

Figure 20. Initiate trust with the secondary ObjectScale

Each of the remote instances will attempt to establish trust, joining the instances in an ObjectScale federation. Initially, the ObjectScale instances are listed as Trust Initiated Trust Status as the federation service on each side tries to communicate to the remote instance over the objectscale-gateway-internal endpoint, and if this system-to-system communication is successful, it moves each remote instance to the Trusted Trust Status.

96 Working with ObjectScale Systems

Add additional ObjectScale instances to an existing ObjectScale federation

About this task

An Administrator user on both ObjectScale systems will need to complete these steps to add additional instances to a federation of ObjectScale systems.

Steps

Complete these steps on the ObjectScale system you wish to make a secondary ObjectScale instance in the federation.

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Systems. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select ObjectScale Systems. For ObjectScale Portal UI: Click the ObjectScale Systems tab. A user can see a list of ObjectScale Systems that the user is authorized to view.

3. Click Join Federation and download the federation signing request file, FederationSigning_ .xml.

Ensure that this file is accessible by both ObjectScale instances.

Complete these steps on the primary ObjectScale instance in the federation.

4. Log in to the ObjectScale Portal UI or VMware vSphere Client.

5. Go to the ObjectScale Systems. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select ObjectScale Systems. For ObjectScale Portal UI: Click the ObjectScale Systems tab. A user can see a list of ObjectScale Systems that the user is authorized to view.

6. Click New Remote Instances and upload the federation signing request file from the other ObjectScale instance. After uploading the xml file, click SAVE.

Figure 21. New Remote Instance

Once you have uploaded this file, the primary ObjectScale instance automatically generates and downloads the signed request file, PostFederationSigning_ .xml. Additionally, the Secondary ObjectScale instance will now appear as a Not Trusted remote instance of ObjectScale within the federation managed by the primary ObjectScale instance.

7. From the secondary ObjectScale instance, click New Remote Instances and upload the signed federation signing request file from the Primary ObjectScale instance. The Primary ObjectScale instance will now appear as a Not Trusted remote instance of ObjectScale within the federation.

8. Finally, to complete the federation process liking these two ObjectScale instances, establish trust between the two ObjectScale instances.

Working with ObjectScale Systems 97

a. From the Secondary ObjectScale instance, select the checkbox of the primary ObjectScale instance that needs to establish trust with the Secondary ObjectScale instance and click Initiate Trust.

The Initiate Trust window appears.

b. Verify that the correct details are shown for the selected ObjectScale instance you wish to establish trust with in the federation and click Yes.

Figure 22. Initiate trust with the primary ObjectScale

c. From the primary ObjectScale instance, select the checkbox of the Secondary ObjectScale instance that needs to establish trust with the primary ObjectScale instance and click Initiate Trust.

d. Verify that the correct details are shown for the selected ObjectScale instance you wish to establish trust with in the federation and click Yes.

Figure 23. Initiate trust with the secondary ObjectScale

Each of the remote instances will attempt to establish trust, joining the instances in an ObjectScale federation. Initially, the ObjectScale instances are listed as Trust Initiated Trust Status as the federation service on each side tries to communicate to the remote instance over the objectscale-gateway-internal endpoint, and if this system-to-system communication is successful, it moves each remote instance to the Trusted Trust Status.

98 Working with ObjectScale Systems

Working with ObjectScale Settings This chapter contains:

Topics:

About Settings About ObjectScale upgrades About SupportAssist Apply the ObjectScale license SAML Service Provider Metadata Manage ObjectScale certificates

About Settings The Settings tab consists of the ObjectScale settings options that you can view and configure.

The Settings tab includes:

Upgrades SupportAssist Licensing SAML Service Provider Metadata ObjectScale Certificate

About ObjectScale upgrades Use this section to understand and complete upgrades to newer versions of Dell EMC ObjectScale, it's components, and object stores.

Overview of ObjectScale upgrades

To upgrade ObjectScale you must first upgrade to a newer version of the ObjectScale components.

NOTE: Currently, ObjectScale upgrades on a three-node cluster are disruptive and require a manual workaround. Before

you begin, see https://dell.com/support/objectscale and search for the KB with the required manual steps you must

perform before and after an upgrade of ObjectScale on a three-node cluster.

For vSphere deployments, use the ObjectScale Portal UI embedded within the vSphere Client to complete this task. Refer to Upgrade to a new version of Dell EMC ObjectScale service for vSphere for details.

For OpenShift deployments, use helm commands to upgrade to the newer version. Refer to Install a new version of ObjectScale for OpenShift for details.

After completing the upgrade to a new version of ObjectScale, you can then use the one-click upgrade process within the ObjectScale Portal UI to upgrade the remaining ObjectScale components and object stores.

Refer to Upgrade ObjectScale components for details on upgrading components and Upgrade an object store using helm for details on upgrading object stores.

Ensure that you upgrade these in the following order:

1. logging-injector 2. objectscale-manager 3. SupportAssist 4. decks

7

Working with ObjectScale Settings 99

5. kahm 6. Object store(s), using either the ObjectScale Portal UI or helm, depending on how the object store is managed.

NOTE: If an upgrade fails and you need to rollback to a previous version, contact Dell support.

Upgrading ObjectScale best practices

Review and follow these best practices when upgrading ObjectScale, ObjectScale components, and object stores.

Do not: Do:

Manually edit pod image tags - they will get automatically replaced in most cases by one of the Kubernetes controllers.

Use the UI to upgrade your apps, or in OpenShift use helm upgrade to change specs and versions on components.

Use helm for components installed through UI (install- controller).

Use the UI for anything installed through the UI, or with a Managed-by: ObjectScale

Manually change the versions of components in Application resources.

Upgrade through the Settings > Upgrades panel once a new yaml is applied to the supervisor.

Delete deployments, especially graphql and install- controller.

Remove applications by following the ObjectScale uninstallation process.

Upgrade object stores before objectscale-manager Upgrade the base components first and then proceed to upgrade object stores.

Force upgrades by manually changing the application version or overriding the pre-update check status.

Check pre-update check results and fix any issues that come up before proceeding with upgrade if possible.

Replace only the install-controller deployment to get new upgrade versions.

Install and apply a new ObjectScale yaml to get a new install-controller, portal, and graphql deployment. New versions will become available as soon as installer is running.

Downgrade ObjectScale. This is not supported. Contact Dell Support to help remedy blocking upgrade issues.

To view the hotfix/version information from using kubectl use, kubectl get app

NAME TYPE VERSION OWNER READY AGE cota objectstore 1.0.0 86m decks decks 2.0.0 12d dellemc-objectscale-license dellemc-license 2.0.0 12d kahm kahm 2.0.0 12d logging-injector logging-injector 1.0.0 12d objectscale-manager objectscale-manager 1.0.0 12d objectscale-monitoring 12d supportassist-objectscale supportassist-objectscale 2.0.0 12d

or helm list

NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION cota svc-objectscale-domain-c10 1 2022-02-2... deployed ecs-cluster-1.0.0 1.0.0 decks svc-objectscale-domain-c10 1 2022-02-1... deployed decks-2.0.0 2.0.0 dellemc-objectscale-license svc-objectscale-domain-c10 1 2022-02-1... deployed dellemc-license-2.0.0 2.0.0 kahm svc-objectscale-domain-c10 1 2022-02-1... deployed kahm-2.0.0 2.0.0 logging-injector svc-objectscale-domain-c10 1 2022-02-1... deployed logging-injector-1.0.0 1.0.0 objectscale-manager svc-objectscale-domain-c10 1 2022-02-1... deployed objectscale-manager-1.0.0 1.0.0 pflex svc-objectscale-domain-c10 1 2022-02-1... deployed service-pod-2.1.0-1610 2.1.0-1610 supportassist-objectscale svc-objectscale-domain-c10 3 2022-02-1... deployed supportassist-2.0.0 2.0.0

100 Working with ObjectScale Settings

In both cases the VERSION and APP VERSION show the installed version of any application, and in a hotfix case would show chartName chartVersion-hotfix.### . For example, objectscale-manager 1.0.0-hotfix.789. Use grep to trim the output on -hotfix to reduce the returned lists to the hotfix versions.

Upgrade to a new version of ObjectScale

Upgrade to a new version of Dell EMC ObjectScale service for vSphere

To prepare for the upgrade ObjectScale to the latest code version to get access to the most current set of product and security features and other functionality you need to complete these steps.

Prerequisites

Before you begin, download the ObjectScale service YAML file.

To download the ObjectScale YAML file, go to VMware JFROG partner site and select Dell > ObjectScale > SupervisorService > . Download the ObjectScale SupervisorService YAML file, objectscale- -vsphere-service.yaml.

Steps

1. Log in to vCenter as Administrator.

2. Go to vSphere Client > Workload Management and click the Services tab. The vSphere Services page appears.

3. In the Dell EMC ObjectScale tile, click ACTIONS and select Add New Version.

4. To register a new version of the service, click UPLOAD and browse to and select the new objectscale_ -vsphere-service.yaml file for vSphere 7.0 U3c. The ObjectScale service details are now shown on the Register Service page.

5. Review the service details and click Next.

6. Accept the terms of the Dell EMC End User License Agreement and click Finish to add the new version of the ObjectScale service to vSphere.

7. Go to Inventory > Workload Cluster > vSphere Services > Overview > Installed.

8. Select Dell EMC ObjectScale service in the vSphere Services table and click EDIT.

9. Complete the edit service wizard that appears.

a. Select the new version of the Dell EMC ObjectScale service. b. Type registry details required to upgrade ObjectScale.

NOTE: Take care to type the correct values into the Repository endpoint, Username, and Passwd fields. If any of

these values are mistyped VMware vSphere will fail to reach the ObjectScale deployment files and the vSphere will

repeatedly attempt and fail to deploy the ObjectScale manager, KAHM, and Decks pods. If this occurs, go to the

vSphere Service page, select the > ACTIONS > EDIT on the Dell EMC ObjectScale tile. The service will be deleted.

i. registryName: Type the URL for the repository endpoint that contains the ObjectScale installation package(s). To use the Dell EMC ObjectScale repository, type: index.docker.io/objectscale If you are upgrading ObjectScale in an environment without external connectivity or are unable to access Docker

for any other reason, enter your private Docker registry address:port which you set up in preparation for the dark site installation method.

ii. regustryUsername: Type the username for the repository endpoint. iii. regustryPasswdType the password for the repository endpoint.

c. Click OK.

After Clicking OK the ObjectScale primary components are upgraded

10. Go to vSphere Client > Workload Management and click the Supervisor Clusters tab to monitor the ObjectScale install and startup.

11. Once the new version of ObjectScale service is installed, refresh the UI to load the new ObjectScale instance. Continue to Upgrade ObjectScale components to upgrade the ObjectScale components and object stores.

Working with ObjectScale Settings 101

Install a new version of ObjectScale for OpenShift

When upgrading to a new version of ObjectScale, you must first install the new versions of the ObjectScale Manager components. After installing these components, you will then be able to install remaining ObjectScale components and Object Stores from the ObjectScale Portal UI.

About this task

In OpenShift deployments, the ObjectScale plugin is upgraded using the helm command. The ObjectScale plugin installs objectscale-portal, objectscale-graphql, and install-controller. Do not modify any resources of the plugin directly using kubectl edit after the helm installation, otherwise, during the helm upgrade those settings will be wiped out.

Steps

1. Download newer versions of the ObjectScale charts from https://www.dell.com/support/home/en-us/product-support/ product/objectscale/drivers.

After downloading the charts, extract the objectscale-portal chart.

2. Get all the current custom values used for the ObjectScale release (objs, in the example) and save it into custom- portal-values.yaml file

helm get -n values objs -o yaml > custom-portal-values.yaml

3. Upgrading to the new version of the ObjectScale release (objs, in the command below).

NOTE: If you have modified the service type = LoadBalancer manually after the helm installation, then the

upgrade will fail as the helm assumes the service was supposed to be ClusterIP.

helm upgrade -n objs /objectscale-portal- .tgz --version -f custom-portal-values.yaml

4. Verify that the upgrade to the new ObjectScale release was successful.

Ensure that the ObjectScale release APP VERSION shows the newly installed version and STATUS shows deployed.

helm -n list

NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION objs default 1 2022-03-2... deployed objectscale-porta... 1.0.1 decks default 1 2022-03-2... deployed decks-2.0.0 2.0.0 dellemc-objectscale-l... default 1 2022-03-2... deployed dellemc-license-2... 2.0.0 kahm default 1 2022-03-2... deployed kahm-2.0.0 2.0.0 logging-injector default 1 2022-03-2... deployed logging-injector-... 1.0.0 objectscale-manager default 1 2022-03-2... deployed objectscale-manag... 1.0.0

Results

You have successfully installed the new versions of the ObjectScale Manager. Now you can continue the upgrade the ObjectScale components and object stores using the ObjectScale Portal UI.

Upgrade ObjectScale components

Perform an upgrade of the ObjectScale components to ensure the ObjectScale components remain up-to-date. After upgrading the ObjectScale components, you can then upgrade the ObjectScale object stores.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Settings. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Settings. For ObjectScale Portal UI: Click the Settings tab. A user can see the ObjectScale Settings sections.

3. Click Upgrades.

102 Working with ObjectScale Settings

The ObjectScale Upgrades section appears showing the COMPONENTS tab, which lists the ObjectScale components, by default.

4. Select all of the ObjectScale components and click PRE-UPGRADE HEALTH CHECK to check the components for upgrade readiness.

The Last Health Check column will provide a pass or fail status. If any of the component health checks failed, you must remedy any issues with the component prior to performing the upgrade.

5. After successfully checking the components for upgrade readiness, deselect all components and then select the logging- injector ObjectScale component. Click UPGRADE to begin the upgrade of the component.

NOTE: Do not select All or some combination of ObjectScale components and perform the upgrade.

6. Optional: Refresh the UI to observe the various states of Status for the component during the upgrade process. The process is complete when the status of the component becomes Success.

7. Repeat steps 5 and 6 for the remaining ObjectScale components in the following order, one component at a time:

a. objectscale-manager b. supportassist-objectscale c. decks d. kahm

Results

You have successfully upgraded all of the ObjectScale components, next follow Upgrading object stores to upgrade the object store(s).

Upgrading object stores

The process to upgrade object stores to a newer version differs depending on how the object store is managed.

How an object store was originally created will determine how you must upgrade that object store to new versions. After upgrading ObjectScale and the other components (KAHM, DECKS, logging-injector, etc) you can now upgrade the object stores within that ObjectScale instance.

For object stores created with the ObjectScale Portal UI follow Upgrade an object store using the ObjectScale Portal UI.

For object stores created with helm follow Upgrade an object store using helm.

Upgrade an object store using the ObjectScale Portal UI

Prerequisites

Ensure that you have upgraded all of the ObjectScale components, prior to using this task to upgrade the object stores managed by the UI.

If you have created object stores in end-user namespaces, on a Linux workstation with kubectl and authenticated to the kubernetes cluster running ObjectScale run these commands prior to using the pre-upgrade health check on the object store(s).

1. Issue this command:

kubectl create clusterrole objectstore-health --verb=get,list,watch -- resource=services,serviceaccounts,secrets

2. Then, for each object store in the end-user namespaces:

kubectl create clusterrolebinding - objectstore-health --clusterrole=objectstore-health -- serviceaccount= : -healthchecks

About this task

Working with ObjectScale Settings 103

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Settings. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Settings. For ObjectScale Portal UI: Click the Settings tab. A user can see the ObjectScale Settings sections.

3. Click Upgrades > OBJECT STORES.

4. Select objectscale-manager component and click PRE-UPGRADE HEALTH CHECK to check the component for upgrade readiness.

5. After the health check completes successfully, click OBJECT STORES.

6. Review the table showing the various object stores within the instance of ObjectScale.

7. Select an object store from the table.

8. Click PRE-UPGRADE HEALTH CHECK to check the object store for upgrade readiness.

9. Click UPGRADE to begin the upgrade of the object store.

10. Optional: Refresh the UI to observe the various states of Status for the object store during the upgrade process. The process is complete when the status of the object store becomes Success.

Upgrade an object store using helm

Object stores originally created using helm cannot be upgrade using the ObjectScale Portal UI and must be upgraded using helm.

About this task

Refer to https://dell.com/support/objectscale for the KB on managing object stores using helm.

About SupportAssist SupportAssist provides a network based connection to Dell Support. SupportAssist enables Dell Support to receive telemetry and issues, events, and alerts from your ObjectScale instance, and to perform remote troubleshooting, resulting in a fast and efficient time to resolution.

NOTE: Dell EMC strongly recommends that you enable the SupportAssist feature to accelerate problem diagnosis, perform

troubleshooting, and help speed time to resolution. If you do not enable the SupportAssist feature, you may need to collect

appliance information manually to assist Dell EMC Support with troubleshooting and resolving problems with your appliance.

The SupportAssist feature employs multiple security layers throughout each step in the remote connectivity process to ensure that you and Dell EMC can use the solution with confidence:

All notifications to Dell EMC originate from your site - never from an outside source - and are kept secure through the use of Advanced Encryption Standard (AES)-256 bit encryption.

IP-based architecture integrates with your existing infrastructure and maintains the security of your environment. Communications between your site and Dell EMC are bilaterally authenticated using RSA digital certificates. Only authorized Dell EMC Customer Service professionals verified through two-factor authentication can download the

digital certificates needed to view a notification from your site. The optional SupportAssist v3 Policy Manager application enables you to grant or restrict Dell EMC Support access based on

your own unique guidelines and requirements, and includes a detailed audit log

(Optional) Configure SupportAssist

You must obtain an access key and pin from Dell EMC in order to configure SupportAssist for the first time. This access key and pin will ensure the accuracy of contact and other customer values and access to Dell Support.

Prerequisites

1. For SupportAssist connectivity you can connect directly with access to FQDN: esrs3-core.emc.com:443.

2. You are logged into the ObjectScale Portal UI.

104 Working with ObjectScale Settings

3. You have applied a valid license to the ObjectScale instance. 4. You are an active Dell EMC customer with login access to https://www.dell.com/support/home/. 5. You must have an access key and pin in order to enable SupportAssist.

To obtain an access key and pin, go to https://www.dell.com/support/home/en-us/product-support/product/objectscale/ overview and click Generate Access key. After completing the required form, Dell EMC will send an email to the email address they have set up in for the Dell portal login. The email will be from the "Dell | ServicesConnectivity Team" and contains the site ID, access key, and pin for the selected customer.

NOTE: The generated access key is valid for seven days.

6. See "SupportAssist port requirements" listed in the ObjectScale Adminstration Guide and validate that required ports are configured properly prior to configuring SupportAssist.

7. A Dell EMC gateway server must already configured on site if you are planning on connecting via a Gateway Server. (SRS Gateway, SAE Gateway or Secure Connect Gateway (SCG))

Steps

1. Go to the ObjectScale Settings. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Settings. For ObjectScale Portal UI: Click the Settings tab. A user can see the ObjectScale Settings sections.

2. Click the SupportAssist tab.

3. Accept the Connect to SupportAssist End User License Agreement.

4. On the Connect to SupportAssist page, select the Select Connection Type to determine which SupportAssist connectivity type will be deployed: Select Connect Directly to connect ObjectScale directly to Dell EMC.

NOTE: Dell EMC highly recommends deploying ObjectScale SupportAssist using a SRS, SAE or SCG gateway, due to

direct connecting to Dell EMC does not support the Remote Support feature.

Select Connect via Gateway Server to connect ObjectScale to Dell EMC through a gateway server.

You must add the Priority, Gateway IP/Host, and Gateway Port values for the gateway server in the Connect via Gateway Server option.

5. If connected via a Gateway Service then, on the Connect to SupportAssist page, enable Remote Support to allow authorized Remote Support engineers to troubleshoot this configuration. Otherwise, leave Remote Support disabled and go to the next step.

6. Configure the Access Key & PIN SupportAssist values for SiteID, Access Key, and PIN.

7. Select the Support Contacts tab to add existing Primary or Secondary contacts.

a. Provide the listed values for the desired Primary contact.

First Name Last Name Email address Phone number Preferred Language

b. Click Add Secondary Contact and provide the required values needed to configure the contact.

8. Click Apply.

9. In the Connection tab verify that the SupportAssist connection was successful and matches the example.

Figure 24. Successfully configure SupportAssist

Working with ObjectScale Settings 105

If a Failed Status is shown, view the Status Message to determine the failure details.

10. Click "Test Connectivity" to validate the SupportAssist connect. When successful, the value shown in the Last Connected column is updated with a newer date and time.

As required, use the panel refresh icon to update the screen prior to automatic updates.

SupportAssist port requirements

Dedicated SupportAssist ports required for ObjectScale SupportAssist and other network traffic.

Port Protocol Direction Description

22 TCP Inbound from SRS Gateway to ObjectScale SSH Secure Copy (SCP) Secure File Transfer

Protocol (SFTP)

9443 TCP Outbound from ObjectScale to SRS Gateway SRS V3 Gateway or later

443 TCP Outbound from ObjectScale to Direct Connect SRS Direct Connect

8443 TCP Outbound from ObjectScale to Direct Connect SRS Direct Connect

8443 TCP Inbound from SRS Gateway to ObjectScale SRS V3 Gateway or later

Edit SupportAssist settings

Add, modify, or remove SupportAssist settings to ensure accurate contact and other customer values.

Prerequisites

You are logged in to ObjectScale and SupportAssist has been previously configured to run on ObjectScale.

Steps

1. Go to the ObjectScale Settings. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Settings. For ObjectScale Portal UI: Click the Settings tab. A user can see the ObjectScale Settings sections.

2. Click SupportAssist.

3. To modify SupportAssist settings:

Click Connection tab to view the current SupportAssist settings. On this tab, you can: Click EDIT to modify the SupportAssist configuration, including connection type, Remote Support, site ID, Access

Key, PIN, or Support Contacts. Click DISABLE to disable the current SupportAssist configuration.

Click Support Contacts to modify or add contacts using the EDIT and DELETE buttons. Click Advanced to modify the system mode, enable Automatic Support Requests, or to re-authenticate SupportAssist.

When modifying the system mode, you are able to set the ObjectScale instance to PreProduction, Normal, or Maintenance.

Disable SupportAssist

Disable SupportAssist for the ObjectScale instance.

Prerequisites

You are logged in to ObjectScale and SupportAssist has been previously configured to run on ObjectScale.

106 Working with ObjectScale Settings

Steps

1. Go to the ObjectScale Settings. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Settings. For ObjectScale Portal UI: Click the Settings tab. A user can see the ObjectScale Settings sections.

2. Click Disable to disable SupportAssit services for this ObjectScale.

Remove SupportAssist

Remove the SupportAssist software on ObjectScale.

Prerequisites

NOTE: After you remove SupportAssist, you have to create a PIN and obtain a new access key.

You are logged in to ObjectScale, and SupportAssist has been previously configured to run on ObjectScale.

Steps

1. Go to the ObjectScale Settings. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Settings. For ObjectScale Portal UI: Click the Settings tab. A user can see the ObjectScale Settings sections.

2. Select DELETE to remove SupportAssist services from this ObjectScale.

3. Click Apply.

Apply the ObjectScale license To activate ObjectScale, apply the license file into the ObjectScale instance.

About this task

Activating ObjectScale with a valid license allows you to create object stores. Subscription and Permanent licenses allow you to create object stores with an overall capacity greater than 30TiB, within the

licensed capacity. The Community Edition license allows you to create object stores up to an overall capacity no larger than 30 TiB, and limits

SupportAssist features.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Settings. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Settings. For ObjectScale Portal UI: Click the Settings tab. A user can see the ObjectScale Settings sections.

3. Select Licensing.

4. Click Apply.

The Apply License box appears.

5. Click Select to browse and upload the ObjectScale license file. Once uploaded, click Apply.

6. Expand the license in the Licensing table to display details about the ObjectScale license and its enabled features and capacities.

Working with ObjectScale Settings 107

SAML Service Provider Metadata Security Assertion Markup Language (SAML) is an open standard for exchanging authentication and authorization data between parties, in particular, between an identity provider and a service provider.

Java Key Store The Java Key Store containing keys required to log in to SAML Provider

Key Alias The Key Alias for the key for SAML Service Provider

Key Password Password for Key Store for SAML Service Provider

DNS Base URL The DNS Base URL required to connect to the SAML Provider Server

Generate SAML Service Provider Metadata

You can use this interface to generate ObjectScale metadata XML to configure ObjectScale trust relationship with the identity provider. The generation requires a java key store and a DNS-domain-name which will be used as the entity Base URL to set the Location in the Assertion Consumer Service.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Settings. For VMware vSphere Client: Go to the Inventory view and select the cluster that is configured for Workload

Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Settings. For ObjectScale Portal UI: Click the Settings tab. A user can see the ObjectScale Settings sections.

3. Click SAML Service Provider Metadata.

4. Click Choose to select a Java Key Store.

5. Enter the details in the Key Alias, Key Password, DNS Base URL fields.

6. Click GENERATE.

Next steps

If you need to delete the SAML Service Provider Metadata, click DELETE METADATA.

If you need to download this SAML Service Provider Metadata, click DOWNLOAD METADATA.

Manage ObjectScale certificates An ObjectScale instance is protected with the by a system-wide SSl certificate.

The following three services must be protected using the SSL certificate:

S3: Used to create, update, and delete S3 objects and buckets using the S3 protocol. Management-gateway: Used to manage the object stores. ObjectScale-gateway: Used to expose for customer-facing ObjectScale services such as Identity and Access Management

(IAM), Federation, and DCM.

You can specify the certificate type. Either the Kubernetes Signed Certificate, which is the default certificate type when you install the ObjectScale with UI, or you can replace this certificate with your own certificate.

To view/manage certificates for the ObjectScale instance, go to Settings > Certificates.

For more information on managing ObjectScale certificates, see "Certificate management and rotation" in the ObjectScale Security Configuration Guide.

108 Working with ObjectScale Settings

Accessing data with IAM and S3 Identity and Access Management (IAM) enables you to have fine-grained access to the ObjectScale S3 resources securely. This functionality ensures that each access request to an ObjectScale resource is identified, authenticated, and authorized.

ObjectScale IAM allows you to add users, roles, and groups.

You can also grant and restrict the access by adding policies to the ObjectScale IAM entities.

This section details the protocols supported by ObjectScale for end-user access to ObjectScale object storage

Topics:

ObjectScale Management REST API introduction Introduction to Identity and Access Management Amazon S3 API support in ObjectScale

ObjectScale Management REST API introduction You can configure and manage certain aspects of ObjectScale using the ObjectScale REST Management API.

For more information about the ObjectScale Management REST API, see these topics:

ObjectScale Management REST API summary Authenticate with the ObjectScale Management REST API

In addition, review the ObjectScale REST API Reference Guide which is auto-generated from the source code and provides a reference for the methods available in the API.

ObjectScale Management REST API summary

The ObjectScale Management REST API enables ObjectScale and/or object store to be configured and managed.

Table 26. ObjectScale-level, Object Store-level, and Object Service APIs - methods summary

API Area Description

ObjectScale Management APIs

Management APIs for creating and managing objectscale-level management users.

Federation APIs for creating and managing objectscale-level federation configurations.

IAM APIs to manage IAM users, roles, and policies within an account.

STS APIs to assume a role and obtain temporary access credentials.

Object Store Management APIs

Metering API for getting various object store metrics, including bucket info, replication status, and performance.

Recovery API to retrieve recovery status for a device or partition.

Replication Control API to control ObjectScale replication. This API allows you to pause, suspend, resume, and throttle replication.

Replication Info API to retrieve ObjectScale replication information for object stores.

Events API to retrieve audit events for a specified namespace.

Account Tenant API for provisioning and managing tenant.

8

Accessing data with IAM and S3 109

Table 26. ObjectScale-level, Object Store-level, and Object Service APIs - methods summary (continued)

API Area Description

Bucket API for provisioning and managing buckets.

Object Store Object Service API

Amazon S3 APIs API to manage Amazon S3.

Download and set up CURL

Steps

1. Download and install CURL: CURL for Linux: https://curl.haxx.se CURL for Windows: https://github.com/curl/curl-for-win

2. Extract contents and append PATH to include location to curl_directory_name/bin.

3. Verify curl functionality with curl -h , for help, or curl -V, for version.

Authenticate with the ObjectScale Management REST API

ObjectScale uses a token-based authentication system for REST API calls. This section provides examples of authenticating with the ObjectScale Management API.

When you are authenticated by ObjectScale, the login API returns an authentication token. You can use this token for authentication in subsequent calls.

For the scoping of the ObjectScale-level and object store-level APIs, see ObjectScale Management REST API summary.

Log in and obtain the auth token for the ObjectScale-level APIs

Use CURL for Windows or Linux to log in to ObjectScale and use ObjectScale-level APIs.

Steps

1. Obtain the ObjectScale Gateway endpoint IP to log in to the ObjectScale-level API

All ObjectScale-level login and API request needs to be made to the ObjectScale gateway loadbalancer endpoint.

OBJECTSCALE_GATEWAY_ENDPOINT=$(kubectl get svc -n $objectscaleNamespace| awk '/ objectscale-gateway[^-]/{print $4}')

2. Log in with the UserID and Password are of an ObjectScale Management User.

curl -vk https:// :443/mgmt/login

The output will end with a section containing the Authentication Token.

< HTTP/1.1 200 OK < Date: Thu, 18 Mar 2021 17:31:40 GMT < Content-Type: application/xml < Content-Length: 93 < Connection: keep-alive < X-SDS-AUTH-TOKEN: OSTOKEN- CiRiOTU4OGNlNS01YjU2LTQ2NmItOGJkYy0wZGYyY2YyZWVlNTISBHJvb3QYqKuSs4QvKKiI7rSELw==.UTv4r +8fu50ndYjT5R4q5KuGMEz9mEneJooc1QAAhPdfCMVfmCScApk2VDN0TYnwqn3BzPpl+cQPGCCVP7nhH3o+gIF s0amBCgIqAfT18PzeRrs42C/ UFJxpWafU9HuHwZe2ACya8aHWcjvsGghZhRsYUyVdKtngT97AUSB+UrAtaA4KBK8PPVplWh3WSVJKJ66UO4wEQ g8GVHHXyv082PvPpxQTc5QpXXSiQqU4LxQncVNpdbACdHSuW5pJX9sNiIKALIHfew7MM4AFqn/ J9lzkfn8Rz9xDSs+bF9TbIQ2qRSEk+w3D7PFipYnkA/g6LKKIQt2o3uU5gq340kgrFQ== < X-SDS-AUTH-USERNAME: root <

110 Accessing data with IAM and S3

root

3. Copy the generated token so you can add it to each command using the -H "Auth_Token" syntax.

You can use the following command to save the Auth Token into "token".

token=$(curl -ik -u root:ChangeMe https://$OBJECTSCALE_GATEWAY_ENDPOINT:443/mgmt/ login | awk '/X-SDS-AUTH-TOKEN/{print $2; exit}')

4. Remove any cr/lf byt running the following command.

token=${token//[$'\r\n']}

You can now run commands and add the --H "X-SDS-AUTH-TOKEN:$token" to pass the Auth Token when issuing ObjectScale-level APIs.

Log in and obtain the auth token for the object store-level APIs

Use CURL for Windows or Linux to log in to ObjectScale and use object store-level APIs.

Prerequisites

About this task

Steps

1. Obtain the ObjectScale Gateway endpoint IP to log in to the ObjectScale-level API

All ObjectScale-level login and API request needs to be made to the ObjectScale gateway loadbalancer endpoint.

OBJECTSCALE_GATEWAY_ENDPOINT=$(kubectl get svc -n $objectscaleNamespace| awk '/ objectscale-gateway[^-]/{print $4}')

2. Log in with the UserID and Password are of an ObjectScale Management User.

curl -vk https:// :443/mgmt/login

The output will end with a section containing the Authentication Token.

< HTTP/1.1 200 OK < Date: Thu, 18 Mar 2021 17:31:40 GMT < Content-Type: application/xml < Content-Length: 93 < Connection: keep-alive < X-SDS-AUTH-TOKEN: OSTOKEN- CiRiOTU4OGNlNS01YjU2LTQ2NmItOGJkYy0wZGYyY2YyZWVlNTISBHJvb3QYqKuSs4QvKKiI7rSELw==.UTv4r +8fu50ndYjT5R4q5KuGMEz9mEneJooc1QAAhPdfCMVfmCScApk2VDN0TYnwqn3BzPpl+cQPGCCVP7nhH3o+gIF s0amBCgIqAfT18PzeRrs42C/ UFJxpWafU9HuHwZe2ACya8aHWcjvsGghZhRsYUyVdKtngT97AUSB+UrAtaA4KBK8PPVplWh3WSVJKJ66UO4wEQ g8GVHHXyv082PvPpxQTc5QpXXSiQqU4LxQncVNpdbACdHSuW5pJX9sNiIKALIHfew7MM4AFqn/ J9lzkfn8Rz9xDSs+bF9TbIQ2qRSEk+w3D7PFipYnkA/g6LKKIQt2o3uU5gq340kgrFQ== < X-SDS-AUTH-USERNAME: root < root

3. Copy the generated token so you can add it to each command using the -H "Auth_Token" syntax.

Accessing data with IAM and S3 111

You can use the following command to save the Auth Token into "token".

token=$(curl -ik -u root:ChangeMe https://$OBJECTSCALE_GATEWAY_ENDPOINT:443/mgmt/ login | awk '/X-SDS-AUTH-TOKEN/{print $2; exit}')

4. Remove any cr/lf byt running the following command.

token=${token//[$'\r\n']}

You can now run commands and add the --H "X-SDS-AUTH-TOKEN:$token" to pass the Auth Token when issuing ObjectScale-level APIs.

Obtain the endpoint IP for the object store Management Gateway and log in to issue object store-level API calls

5. Assign the object store's name to the variable objectstoreName and assign the object store's namespace to the variable objectstoreNamespace .

OBJECTSTORE_MANAGEMENT_GATEWAY=$(kubectl get svc -n $objectstoreNamespace | grep $objectstoreName | awk '/-management-gateway/{print $4}')

Below is an example of how to use ObjectScale-level token to make and object store API call.

curl -ks -X GET -H "X-SDS-AUTH-TOKEN:$token" -H "Accept:application/xml" -H "X-EMC- Override: true" https://${OBJECTSTORE_MANAGEMENT_GATEWAY}:4443/object/tenants

ObjectScale Management Service

The ObjectScale Management Service manages users and roles and is used for establishing trust with other external identity providers. It provides an API for authentication/authorization that allows for secure token generation which will be accepted by other ObjectScale services.

Management service is a part of the ObjectScale Federation Service. It provides the following functionality:

Defines roles for management users. Supplies the /mgmt APIs.

Provides method to process OSTOKEN correctly for the IAM, Federation Service, and Object Control Service in an object store.

Modifies the IAM and Federation Service client to transparently handle OSTOKEN interactions. 1. A user first logs in to the /mgmt/login endpoint. The returned OSTOKEN will have the roles associated with the user.

2. The user can then present this OSTOKEN (in X-SDS-AUTH-TOKEN header) to request services from IAM, Federation Service, and Object Control in an object store.

3. These ObjectScale services will first authenticate the OSTOKEN with Management service and based on the roles available in the token a determination is made whether the user is authorized to access the requested resource.

Objectscale Token (OSTOKEN) format

OSTOKEN is based on JSON Web Token (JWS) and supports both the symmetric (HS256) and asymmetric (RS256) based tokens. User tokens support logout functionality they will use HS256 based JWS tokens.

For HS256-based tokens, the "kid" header parameter is be used to identify the key used to sign the tokens. For RS256-based tokens, the "x5u" header parameter is used to obtain the certificate/public key that we need to use to verify the token.

OSTOKEN expiration can be controlled by x-dell-token-ttl-mins header in login APIs between 15mins and 8hrs.

The default expiration for /mgmt/login is 15mins.

NOTE: All OSTOKENs are opaque and are intended to be used as is. ObjectScale exposes some APIs to determine the

expiry time of the OSTOKEN.

112 Accessing data with IAM and S3

Logout

The logout API ends a session.

Each user is allowed a maximum of 100 concurrent authentication tokens. Beyond this limit, the system refuses any new connection for a user until tokens free up. Tokens can free up by expiring naturally, or by issuing the following API call:

GET https:// :443/mgmt/logout

If you have multiple sessions running simultaneously, the following API call forces the termination of all tokens related to the current user.

GET https:// :443/mgmt/logout?force=true

The following example shows a logout request. You pass in the authentication token from header or cookie to log out.

GET https:// :443/mgmt/logout

X-SDS-AUTH-TOKEN:{Auth_Token}

The response should be HTTP 200.

Introduction to Identity and Access Management In ObjectScale, Identity and Access Management (IAM) a shared service within a single ObjectScale instance used to manage Accounts and the Account's IAM entities.

IAM provides an AWS-compatible authentication and authorization mechanism that are availed by other ObjectScale services such as:

Datahead (S3) Geoservice Object store management service

In this release the top-most level of the ObjectScale IAM hierarchy is an Account. Several Accounts can be defined in single ObjectScale. When an IAM account is added to an object store, that account becomes a tenant within that object store. A tenant is a logical construct resulting from the binding of the IAM account to the object store.

Every Account has a globally unique identifier assigned to it by the IAM service at the time of creation An IAM Account contains other IAM entities like Users, Groups, Roles, Policies, and Service Providers associated with it. You cannot create or modify an Account to have another Account associated with it.

NOTE: ECS Object users are not supported in ObjectScale.

In ObjectScale, each account consists of replicated IAM entities and ObjectScale local IAM entities. Local IAM entities remain local within the ObjectScale instance and are not replicated. Global entities are replicated to other ObjectScale instances. Replicated IAM entities and ObjectScale local IAM entities have separate APIs.

The ObjectScale instance where the Account was created initially owns that Account and is known as the Account Owner. That account is a primary account on that ObjectScale instance. Within ObjectScale there can be only one Account Owner for any given Account and it's underlying IAM entities.

Multiple ObjectScale instances can be connected to each other forming an ObjectScale Federation. Within this federation all ObjectScale instances have a trust relationship established with each other. Any Federation member "knows" about other Federation members through ObjectScale Federation Service. Any Federation member "knows" about all existing Accounts across Federation, i.e. there is a shared Account Registry. When a primary IAM account is replicated from it's ObjectScale instance to another ObjectScale instance(s) within the federation, it becomes a secondary on these other ObjectScale instances. For more information about federating ObjectScale instances.

An Account can be changed. Meaning that an authorized user can perform CRUD operation on any entities associated with that account. However, such operations will always be performed on Account Owner cluster. When the IAM entity is changed, the effects of those changes may not take effect immediately.

If a user tries to change an Account from an ObjectScale that doesn't own that Account, the user will get HTTP 301 or 308 message along with the URL that corresponds to the Account Owner.

Accessing data with IAM and S3 113

Replicated roles are used to set up bucket replication.

The Dell EMC ObjectScale RestAPI zip file with the supported IAM APIs is available at https://www.dell.com/support/home/ product-support/product/objectscale/drivers.

Account Management

Account Management enables you to manage IAM identities within each account such as users, groups, and roles.

All IAM entities have a unique ID associated with it. Deleting and re-creating an entity with the same name creates a unique ID for the new entity.

An IAM Account contains other IAM entities like Users, Groups, Roles, Policies, and Service Providers associated with it. You cannot create or modify an Account to have another Account associated with it. Each account consists of replicated IAM entities and local IAM entities. Local IAM entities remain local within the ObjectScale instance and are not replicated. Global entities are replicated to other ObjectScale instances. Replicated IAM entities and local IAM entities have separate APIs.

Primary Account

A primary account is used to create management users. This task describes management roles for the user types in an ObjectScale local management account within a single ObjectScale instance.

Table 27. Primary Accounts within ObjectScale

User type Management roles

Security Administrator Manage certificates Administer other management users Manage authentication providers Manage security settings

System Administrator Manage accounts Manage licensing Manage storage Manage monitoring/alerts/audits

System Monitor (read-only access) Manage monitoring, alerts, audits

Account Administrator Manage accounts

IAM Identities

Table 28. Identities

Field Description

Account root user Account root user is an admin user in the account. Only the account root user can access ObjectScale

instance UI. Account root user is the owner of the buckets and objects

that are created by the IAM entities.

IAM user An IAM user is a person or an application in the account that can interact with ObjectScale resources.

An IAM user can belong to one or more IAM groups. It is possible to create, view, modify, delete, and list IAM

users in ObjectScale using both API and UI. IAM users cannot access ObjectScale instance UI.

IAM group An IAM group is a collection of IAM users. IAM groups do not nest and contain only IAM users. IAM groups let you specify permissions for all the users in

the group making management easier.

114 Accessing data with IAM and S3

Table 28. Identities (continued)

Field Description

Creating and managing groups can be done from both UI and API.

Tagging on groups is not supported.

IAM role An IAM role is similar to a user, in that it is an identity with permission policies that determine what the identity can and cannot do.

An IAM role does not have any credentials that are associated with it.

An entity assumes a role by calling an API that provides it with temporary credentials to access a resource.

A federated user can assume an IAM role by authenticating with external identity provider.

An IAM user can assume a role in the same or different account (cross-account access).

NOTE: IAM and account root users access S3 and IAM APIs using Access Keys. Access Keys are long-term credentials

which consists of an access key ID and secret access key. A user can have at most two Access Keys associated with it at

any time.

Tagging IAM Entities (Users and Role)

A tag is a label that you assign to a resource. Each tag consists of a key and an optional value, both of which you define. Custom attributes are added to users and roles using a tag key-value pair. These tags can be used to control the access of an entity to resources or to control what tags can be attached to an entity. Groups and policies cannot be tagged. You can apply the same tag to multiple entities. But multiple tags on one entity cannot have the same key. Fifty tags per IAM entity are allowed.

IAM error codes

Table 29. IAM error codes

Error type HTTP status code Description

AccessDeniedException 400 Indicates that you do not have the required access to perform the action.

ConcurrentModification 409 Indicates that multiple requests are submitted simultaneously to modify the object. You need to wait for a few minutes and submit the request again.

DeleteConflict 409 Indicates that the request is raised to delete a resource that is attached with another entity.

EntityAlreadyExists 409 Indicates that the request is raised to create a resource that already exists.

ExpiredToken 400 Indicates that the Web identity token that is used to perform the action is expired or not valid.

IDPRejectedClaim 403 Indicates that the identity provider (IdP) reported that authentication failed.

InternalFailure 500 Indicates that the request failed due to an unknown error, exception, or failure.

InvalidAction 400 Indicates that the requested action is not valid.

InvalidInput 400 Indicates that an invalid or an out-of-range value is provided for an input.

Accessing data with IAM and S3 115

Table 29. IAM error codes (continued)

Error type HTTP status code Description

InvalidParameterValue 400 Indicates that an invalid or an out-of-range value is provided for an input parameter.

LimitExceeded 409 Indicates that the request is rejected because an attempt is made to create resources beyond the current account limits.

MalformedPolicyDocument 400 Indicates that the provided policy document is malformed.

MissingAction 400 Indicates that the action or a required parameter is missed in the request.

MissingParameter 400 Indicates that the required parameter is missed in the request.

NoSuchEntity 404 Indicates that the referenced entity does not exist.

NotImplemented 501 Indicates that the mentioned functionality is not implemented yet.

PackedPolicyTooLarge 400 Indicates that the total packed size of the session policies and session tags combined is too large.

PermissionDenied 403 Indicates that the principal does not have the required permission to perform the action.

ServiceFailure 500 Indicates that the request is failed because of an unknown error, exception, or failure.

ServiceUnavailable 503 Indicates that the request is failed due to a temporary failure of the server.

ValidationError 400 Indicates that the input fails to satisfy the constraints specified by the specific API.

IAM supported condition keys

IAM supports the following condition keys:

Global condition keys Type Description

aws:CurrentTime Date To check for date and time conditions

aws:EpochTime Date To check for date and time conditions using a date in epoch or UNIX time

aws:PrincipalArn ARN Checks the ARN of the IAM user or role that made the request.

aws:UserAgent String To check the client application of the requestor.

aws:PrincipalTag/ tag-key String Checks that the tag attached to the principal making the request matches the specified key name and value.

aws:RequestTag/ tag-key String Checks that the tag key-value pair is present in an AWS request.

aws:ResourceTag/ tag-key String Checks that the tag key-value pair is attached to the resource.

aws:SourceIp IpAddr To check the IP address of the requester

aws:TagKeys String,

ForAllValues:String

ForAnyValue: String

This context key is a list of tag keys without values

116 Accessing data with IAM and S3

Global condition keys Type Description

aws:TokenIssueTime Date Checks the date and time that temporary security credentials were issued.

aws:principaltype String Indicates the type of principal making the request. Root user is Account. IAM user is User. SAML or Assumed role user is AssumedRole.

aws:userid String Based on authorized user access is set to the following: Root user ARN if root user is requester. IAM user unique id IAM user is requester. If SAML federated user is requester, it is set to the

role-id:caller-specified-role-name If assumed role user is requester, it is set to the

role-id:caller-specified-role-name role-id: is the unique id of role caller-specified-role-name: is the RoleSessionName in AssumeRole request or the name attribute value in SAML assertion passed to AssumeRoleWithSAML request.

aws:username String Based on authorized user access, if requester is an IAM user, it is set to the IAM username otherwise it is not set.

IAM condition keys Type Description

iam:PermissionsBoundary String Checks that the specified policy is attached as permissions boundary on the IAM principal resource.

iam:PolicyARN ARN Checks the ARN of a managed policy in requests that involve a managed policy.

iam:ResourceTag/ key-name String Checks that the tag attached to the IAM entity (user or role) matches the specified key name and value.

STS and SAML condition keys Type Description

saml:aud String An endpoint URL to which SAML assertions are presented. The value for this key comes from the SAML Recipient field in the assertion, not the Audience field.

saml:edupersonorgdn String This is an eduPerson attribute in SAML assertion.

saml:iss String The issuer, which is represented by a URN.

saml:namequalifier String This contains a hash value that represents the combination of the saml:doc and saml:iss values. It is used as a account qualifier; the combination of saml:namequalifier and saml:sub uniquely identifies a user.

saml:sub String This is the subject of the claim, which includes a value that uniquely identifies an individual user within an organization.

saml:sub_type String This key can have the value persistent , transient , or consist of the full Format URI from the Subject and NameID elements used in your SAML assertion. A value of persistent indicates that the value in saml:sub is the same for a user between sessions. If the value is

Accessing data with IAM and S3 117

STS and SAML condition keys Type Description

transient , the user has a different saml:sub value for each session.

S3 condition keys Description

s3:x-amz-acl Specifies the canned ACL in the request.

s3:x-amz-grant- permission Specifies permission for the following access. read write read-acp write-acp full-control

s3:x-amz-copy-source Enables restricting copy source to a specific bucket, folder, or object.

s3:x-amz-metadata-directive Specifies certain behavior to be enforced during object uploads (COPY vs REPLACE).

s3:x-amz-server-side-encryption Specifies that the request should contain this header to ensure that the uploads are stored encrypted.

s3:VersionId Limits access to specific versions of object.

s3:LocationConstraint Using this condition key, you can restrict a user to create a bucket in a specific AWS Region.

s3:delimiter Used to require the requester to specify delimiter parameter.

s3:max-keys Limits ListBucket requests to the set s3:max-keys value.

s3:prefix Limits ListBucket and ListBucketVersions to specific prefix.

s3:ExistingObjectTag/ Using this condition key, you can limit the permission for the s3:PutObjectAcl action to only on objects that have a specific tag key and value.

s3:RequestObjectTagKeys Using this condition key, you can limit permission for the s3:PutObject action by restricting the object tags allowed in the request.

s3:RequestObjectTag/ Using this condition key, you can limit permission for the s3:PutObject action by restricting the object tags allowed in the request.

IAM limitations on entities and objects

IAM has certain limitations on its resources such as naming the entities, characters to be used for the identities, number of policies to be attached to an entity, and the number of resources that can be linked to an entity.

NOTE: Paths are not supported for IAM entities.

IAM entity name limits

Resource Limits

Names of users, groups, roles, and managed policies

Must be unique within the account. Must be alphanumeric and it may include any of these special characters: Plus (+),

equal (=), comma (,), period (.), at (@), underscore (_), and hyphen (-).

NOTE: These names are case insensitive.

Inline policy names Must be unique to the user, group, or to the role that they are embedded in. Can contain any Basic Latin (ASCII) characters except these special characters:

Backward slash (\), forward slash (/), asterisk (*), question mark (?), and space.

118 Accessing data with IAM and S3

Resource Limits

These characters are reserved according to the RFC (Request for Comments) 3986 Internet standard.

Policy documents Can contain these Unicode characters: horizontal tab (U+0009), linefeed (U+000A), carriage return (U+000D), and characters in the range from U+0020 to U+00FF.

IAM entity object limits

Resource Limit

Customer managed policies in an AWS account 500

Groups in an AWS account 100

Roles in an AWS account 200

Managed policies that are attached to an IAM group 10

Managed policies that are attached to an IAM role 10

Managed policies that are attached to an IAM user 10

IAM users in a group Equal to user quota in an account

Users in an account 500

IAM entities limits

Resource Limit

Access keys that are assigned to an IAM user 2

Access keys that are assigned to the account root user 2

Groups an IAM user can be a member of 10

Identity providers (IdPs) associated with an IAM SAML provider object

10

Keys per SAML provider 1

Managed policies attached to an IAM group 10

Permissions boundaries for an IAM user 1

Permissions boundaries for an IAM role 1

SAML providers in an AWS account 10

Tags that can be attached to an IAM user 50

Tags that can be attached to an IAM role 50

Versions of a managed policy that can be stored 5

IAM entity character limits

Description Limit

Path 512 characters

User name 64 characters

Group name 128 characters

Role name 64 characters

Accessing data with IAM and S3 119

Description Limit

Tag key 128 characters

Tag value 256 characters NOTE: Tag values can be empty. That is, tag values can have a length of 0 characters.

Unique IDs created by IAM 128 characters

Policy name 128 characters

Role trust policy JSON text (the policy that determines who is allowed to assume the role)

2,048 characters

Role session name 64 characters

Max role session duration 24 hours

For inline policies You can add as many inline policies as you want to an IAM user, role, or group. But the total aggregate policy size (the sum size of all inline policies) per entity cannot exceed the following limits: User policy size cannot exceed 2,048 characters. Role policy size cannot exceed 10,240 characters. Group policy size cannot exceed 5,120 characters.

NOTE: IAM does not count white space when calculating the size of a policy against these limitations.

For managed policies You can add up to 10 managed policies to an IAM user, role, or group.

The size of each managed policy cannot exceed 6,144 characters.

NOTE: IAM does not count white space when calculating the size of a policy against these limitations.

For session policies You can pass only one JSON policy as a parameter when you programmatically create a temporary session for a role or federated user.

The size of each session policy cannot exceed 2,048 characters.

Access Management

Access is managed by creating policies and attaching them to IAM identities or resources.

ObjectScale IAM protects the following resources:

Object Head API S3 (buckets and objects). See Accessing data with IAM and S3 for details on the supported IAM S3 APIs.

STS APIs AssumeRole (Provides temporary credentials for cross account access) AssumeRoleWithSAML (Provides temporary credentials for SAML authenticated users)

IAM API

120 Accessing data with IAM and S3

IAM Policies and ACLs

A policy is an object that when associated with an identity or resource defines their permissions. Permissions in the policies determine if the request is permitted or denied.

IAM Policies

ObjectScale IAM enables creation, modification, listing, assigning, and deletion of policies on an identity or resource. IAM policies are stored in JSON format.

Using policies you can:

Specify actions on a resource. Identify resources. Identify principals that are applicable for the policies. Specify conditions that are applicable

IAM policies define permissions for an action regardless of the method that you use to perform the operation. The following policy types, are designed for use in ObjectScale:

Table 30. Policy Types

Policies Description

Identity-based policies Policies that are assigned to users, groups, and roles which grant permissions to an identity. Inline Policies Managed Policies (Both ObjectScale and Customer managed)

Resource-based policies Resource-based policies are inline policies that are assigned to an ObjectScale resource that grants specified principal permission to perform specific action on the resource. Bucket Policy - Tweaked existing support for bucket policies to support IAM use cases. Trust Policy - Is a resource-based policy that is attached to an IAM role. Trust policies

identify the principal entities that can assume the role.

Permission Boundaries Use a managed policy as the permissions boundary for an IAM entity (user or role). That policy defines the maximum permissions that the identity-based policies can grant to an entity, but does not grant permissions. Permissions boundaries do not define the maximum permissions that a resource-based policy can grant to an entity.

Session policies Session policies are used with AssumeRole and AssumeRoleWithSAML APIs. Session policies limit the permissions that the identity-based policies of a role or user grants to the session. Session policies limit permissions for a created session, but do not grant permissions.

Access Control Lists (ACLs) Tweaked existing ObjectScale ACLs on buckets and objects to support IAM use cases. ACLs are cross-account permissions policies that grant permissions to the specified principal. ACLs cannot grant permissions to entities within the same account.

NOTE: If there is an explicit deny in any policy, then the request is denied otherwise there must be a policy that explicitly

allows the request. If neither then by default the request is denied.

Policy Basics

Policy is made up of one or list of statements. A statement is contained within a series of elements.

Version Specify the version of the policy language that you want to use. As a best practice, use the latest 2012-10-17 version.

Statement Use this main policy element as a container for the following elements. You can include more than one statement in a policy.

Sid (Optional) Include an optional statement ID to differentiate between your statements.

Effect Use Allow or Deny to indicate whether the policy allows or denies access.

Principal (Required in only some circumstances) If you create a resource-based policy, you must indicate the account, user, role, or federated user to which you would like to allow or deny access. If you are creating

Accessing data with IAM and S3 121

an IAM permissions policy to attach to a user or role, you cannot include this element. The principal is implied as that user or role.

Action Include a list of actions that the policy allows or denies.

Resource ARN of resources the permission applied on. * apply to all resources.

Condition (Optional) Specify the circumstances under which the policy grants permission.

ACLs

Access control lists allow you to manage access to objects and buckets. An ACL is attached to all objects and buckets.

S3 non-ObjectScale IAM access S3 ObjectScale IAM access

Users own buckets and objects. Buckets are owned by the account to which they belong and objects are owned by the account to which the user that created the object belongs.

Bucket and object owners can be changed. Buckets and object owners can never be changed.

Any user can be a non-group grantee in an ACL. Only a account can be a non-group grantee in an ACL.

S3 request authorization

During the S3 request authorization process, ObjectScale evaluates permission using user, bucket, and object contexts as needed.

Context Description

User In this context, if the requester is an ObjectScale IAM principal, the principal must have permission from the parent account to which it belongs. In this step, the subset of policies that are owned by the parent account (also referred as the context authority) is evaluated. This subset of policies includes the user policy that the parent attaches to the principal. If the parent also owns the resource in the request (bucket, object), then the corresponding resource policies (bucket policy, bucket ACL, and object ACL) are also evaluated at the same time.

Bucket In this context, ObjectScale evaluates policies that are owned by the account that owns the bucket. If the account that owns the object in the request is not same as the bucket owner, in the bucket context the policies are checked to verify that the bucket owner has not explicitly denied access to the object. If there is an explicit deny set on the object, then the request is not authorized.

Object In this context, the requester must have permissions from the object owner to perform a specific object operation. In this step, the object ACL is evaluated if required.

Bucket authorization

In the S3 bucket operation authorization process, at first the system evaluates whether the requester is an IAM user. If yes, then the request is evaluated against the user context and the bucket contexts. If both verifications are authorized, the access is granted. Else, it is denied.

The below table describes the summary of access details for the same and cross account bucket operation:

Bucket owner (account)

Requestor (account, user)

Comments

A1 U1 The user or the bucket policy determines the access. There is no bucket ACL check.

A1 U2 U2 needs IAM policy from A2, if A1 bucket policy does not a make a determination, then the system checks the bucket ACL.

A1 R1 IAM policy is not relevant for root user (R1). If A1 bucket policy does not a make a determination, then the system checks the bucket ACL.

122 Accessing data with IAM and S3

Bucket owner (account)

Requestor (account, user)

Comments

A1 R2 IAM policy is not relevant for root user (R2). If A1 bucket policy does not a make a determination, then the system checks the bucket ACL.

NOTE: In this table, the following legends are used:

A1 = first account, A2 = second account, U1 = user from the first account, U2 = user from the second account, R1 = root user from the first account, and R2 = root user from the second account.

Object authorization

In the S3 object operation authorization process, at first the system evaluates whether the requester is an IAM user. If yes, then the request is evaluated against the user, bucket, and object contexts. If these three contexts verifications are authorized, the access is granted. Else, it is denied.

The below table describes the summary of access details for the same and cross account bucket operation:

Bucket owner (account)

Object owner (account)

Requestor Comments

A1 A1 U1 Access is determined by the user and/or by the bucket policy. No object ACL check

A1 A1 U2 U2 needs IAM policy from A2 and if A1 bucket policy does not a make a determination, then the system checks the object ACL

A1 A1 R1 IAM policy not relevant for R1. If A1 bucket policy does not a make a determination, then the system checks the object ACL

A1 A1 R2 IAM policy not relevant for R2. If A1 bucket policy does not a make a determination, then the system checks the object ACL

A1 A2 U1 U1 needs IAM policy or bucket policy allow. Object ACL must allow A1 access.

A1 A2 U2 U2 needs IAM policy allow. Bucket policy should not deny.

NOTE: Bucket policy cannot allow access.

A1 A2 U3 U3 needs IAM policy allow. Bucket policy should not deny. Object ACL must allow A3 access.

NOTE: Bucket policy cannot allow access.

A1 A2 R1 IAM policy not relevant. Bucket policy should not be deny. Object ACL needs to allow A1 access.

NOTE: Bucket policy cannot allow access.

A1 A2 R2 IAM policy not relevant. Bucket policy should not be deny. Object ACL must allow A2 access.

NOTE: Bucket policy cannot allow access.

A1 A2 R3 IAM policy not relevant. Bucket policy should not be deny. Object ACL must allow A3 access.

NOTE: Bucket policy cannot allow access.

NOTE: In this table, the following legends are used:

A1 = first account, A2 = second account, A3 = third account, U1 = user from the first account, U2 = user from the second account, U3 = user from the third account, R1 = root user from the first account, R2 = root user from the second account, and R3 = root user from the third account.

Accessing data with IAM and S3 123

IAM and STS resource requests

ObjectScale evaluates the authorization requests on ECS IAM and STS resources within one account in the following manner. 1. Deny evaluation - By default, all requests are denied (implicit deny). PEM evaluates all policies within the account that

apply to the request. These include resource-based policies, permissions boundaries, role session policies, and identity-based policies. In all these policies, enforcement code looks for a Deny statement that applies to the request (explicit deny). If the code finds even one explicit deny that applies, the code returns a final decision of Deny. If there is no explicit deny, the evaluation continues.

2. Resource-based policies - If the requested resource has a resource-based policy that allows the principal entity to perform the requested action, then the code returns a final decision of Allow. If there is no resource-based policy, or if the policy does not include an Allow statement, then the code continues. This logic can behave differently if you specify the ARN of an IAM role or user as the principal of the resource-based policy. Someone can use session policies to create a temporary credential session for that role or federated user. In that case, the effective permissions for the session might not exceed those allowed by the identity-based policy of the user or role.

3. IAM permissions boundaries - The enforcement code then checks whether the IAM entity that is used by the principal has a permissions boundary. If the policy that is used to set the permissions boundary does not allow the requested action, then the request is implicitly denied. The code returns a final decision of Deny. If there is no permissions boundary, or if the permissions boundary allows the requested action, the evaluation continues.

4. Session policies - The code then checks whether the principal entity is using a session that was assumed by passing a session policy. You can pass a session policy while using temporary credentials for a role or federated user. If the session policy is present and does not allow the requested action, then the request is implicitly denied. The code returns a final decision of Deny. If there is no session policy, or if the policy allows the requested action, the code continues.

5. Identity-based policies - The code then checks the identity-based policies for the principal entity. For an IAM user, these include user policies and policies from groups to which the user belongs. If any statement in any applicable identity-based policies allows the requested action, then the PEM evaluation returns a final decision of Allow. If there are no statements that allow the requested action, then the request is implicitly denied, and the code returns a final decision of DenyErrors that is any errors that are encountered by PEM during the evaluation will throw an exception and stops evaluation.

Security Token Service

The Security Token Service (STS) enables you to request temporary credentials, for IAM users or for other users that are externally authenticated (SAML).

ObjectScale IAM supports these two STS APIs:

AssumeRole (Provides temporary credentials for cross account access)

AssumeRoleWithSAML (Provides temporary credentials for SAML authenticated users)

NOTE: The temporary credentials from the AssumeRole and AssumeRoleWithSAML APIs consist of an access key ID,

secret access key, and a session token. These temporary credentials cannot be revoked.

Accessing accounts using AssumeRole

AssumeRole returns a set of temporary security credentials that you can use to access IAM and S3 resources.

NOTE: The role trust relationship should grant permission to an entity to assume the role.

Same account access with AssumeRole

You can access the same account using AssumeRole by attaching a policy to the user (identical to the previous user in a different account) or by adding the user as a principal directly in the role trust policy.

Method Example

Attaching a policy to the user 1. Trust policy for Role assumeRoleSameAccount in ns1:

{ "Version": "2012-10-17", "Statement": [ {

124 Accessing data with IAM and S3

Method Example

"Effect": "Allow", "Principal": { "AWS": "urn:ecs:iam::ns1:root" }, "Action": "sts:AssumeRole" } ] }

2. Policy is attached to the user1 in ns1 to AssumeRole:

{ "Version": "2012-10-17", "Statement": [ { "Action": [ "sts:AssumeRole" ], "Resource": "urn:ecs:iam::ns1:role/ assumeRoleSameAccount", "Effect": "Allow", "Sid": "VisualEditor0" } ] }

Adding the user to the role trust policy

Trust policy for Role in ns1 with an ObjectScale IAM user:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "urn:ecs:iam::ns1:user/user1" }, "Action": "sts:AssumeRole" } ] }

Cross account access with AssumeRole

By default, an IAM user in one account has no access to buckets in another account. However, you can access different accounts using the role trust policy through AssumeRole.

Your account must be trusted by the role to assume a role from a different account. The trust relationship is defined in the role trust policy when the role is created. That trust policy states which accounts are allowed to delegate that access to users in the account. Also, ensure that you have permissions that are delegated from the user account administrator. The administrator must attach a policy that allows you to call AssumeRole for the Amazon Resource Name (ARN) of the role in the other account.

For example, your organization has multiple account. From which, you segregate a staging environment from a production environment. Certain users such as developers from the staging account may also want to access the production account when you move the staging environment to the production.

For this scenario, the admin creates two groups for the staging account namely Dev and QE, and each group has its own policy.

In the production account, the administrator performs the following: Specifies a trust policy to the role to state that the staging account as a Principal. So that the authorized users from the

staging account can use that role. Specifies which role users have read and write permissions to the productionsys bucket through a permissions policy. Shares the account and role information with the users who need to assume the role.

In the staging account, the administrator grants permission to the Dev group to assume the UpdateSys role. By doing this, the Dev group members can switch their role to the required and permitted role. For example, the Dev group members can

Accessing data with IAM and S3 125

switch their role to the UpdateSys role in the production account. Other users such as QE group members cannot switch their role. Hence, they cannot access the productionsys bucket.

In this process, STS verifies whether the requester is a trusted entity. After verifying, it returns temporary credentials to the authorized users to perform the required actions.

Example

1. Trust policy for Role in ns1:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "urn:ecs:iam::ns2:root" }, "Action": "sts:AssumeRole" } ] }

2. Policy that is attached to the user in ns2 to AssumeRole:

{ "Version": "2012-10-17", "Statement": [ { "Action": [ "sts:AssumeRole" ], "Resource": "urn:ecs:iam::ns1:role/assumeRoleCrossAccount", "Effect": "Allow", "Sid": "VisualEditor0" } ] }

IAM SAML support

Security Assertion Markup Language (SAML) is an open standard for exchanging authentication and authorization data between parties, in particular, between an identity provider (IdP) and a service provider.

ObjectScale currently supports only SAML integration with Microsoft Active Directory Federation Services (ADFS) version 6.3.0.0. This establishment enables the federated users to access ObjectScale resources.

Setting up a SAML-compliant provider

ObjectScale currently supports only ADFS as a SAML-compliant Identity Provider. Perform the following steps to use ADFS as a SAML-compliant Identity Provider. ObjectScale will be the service provider.

About this task

You can use this interface to generate ObjectScale metadata XML to configure ObjectScale trust relationship with the identity provider. The generation requires a java key store and a DNS-domain-name which will be used as the entity Base URL to set the Location in the Assertion Consumer Service.

Steps

1. Download the Identity Provider (ADFS) metadata file. The default URL to download ADFS metadata is https://[server- name]/FederationMetadata/2007-06/FederationMetadata.xml.

2. Upload the downloaded metadata xml file when creating Identity provider.

3. To create the Identity Provider in ObjectScale Portal UI, follow to New Identity Provider.

In order to establish trust relationship between ObjectScale and ADFS, ObjectScale metadata xml file is required.

126 Accessing data with IAM and S3

4. To create or download the ObjectScale SAML Service Provider Metadata file, refer Generate SAML Service Provider Metadata.

5. Establish trust relationship between ObjectScale and ADFS using the downloaded ObjectScale SAML Service Provider Metadata file.

6. Add claim rules in ADFS to add the required elements such as NameId, RoleSessionName, and Roles to the SAML authentication process.

NOTE: If required, contact Dell remote support for configuring claim rules in ADFS.

NOTE: Only one IdP is supported in the federation metadata from ADFS.

AssumeRoleWithSAML

In order to use AssumeRoleWithSAML, you must configure your SAML identity provider (IdP) like ADFS to issue the claims required by ObjectScale. IAM role must be created that specifies this SAML Provider in the trust policy. In order to use AssumeRoleWithSAML from each ObjectScale instance, you must first setup Relying Party Trust with that

ObjectScale service provider metadata and get the SAML Token from that specific relying party trust. AssumeRoleWithSAML returns a set of temporary security credentials for users who have been authenticated through a

SAML authentication response. This operation provides a mechanism for tying an enterprise identity store or directory to role-based access without

user-specific credentials or configuration. Calling AssumeRoleWithSAML does not require the use of ObjectScale security credentials. The identity of the caller is

validated by the claims that are provided in the SAML Assertions by the identity provider. Temporary credentials consist of an access key ID, a secret access key, and a security token. Following condition keys are supported in the AssumeRolePolicyDocument.

saml:aud saml:iss saml:sub saml:sub_type saml:edupersonorgdn saml:namequalifier

Example role trust policy

{ "Version":"2012-10-17", "Statement":[ { "Effect":"Allow", "Principal":{ "Federated":"urn:aws:iam::s3:saml-provider/provider1" }, "Action":"sts:AssumeRoleWithSAML", "Condition":{ "StringEquals":{ "SAML:sub":"ADFS\\Bob", "SAML:aud":"https://10.247.179.105/saml", "SAML:eduPersonOrgDN":[ "ObjectScale" ], "SAML:iss":"http://AD.adfs.emc.com/adfs/services/trust" } } } ] }

Accessing data with IAM and S3 127

Attributes in SAML assertion

The following attributes are required in SAML assertion.

https://aws.amazon.com/SAML/Attributes/RoleSessionName https://aws.amazon.com/SAML/Attributes/Role

NOTE:

The Role attribute must be of the format SAML Provider URN, Role URN to be used from ObjectScale for an AD

Group.

If you must use saml:edupersonorgdn, then oid attribute must also be present in the SAML assertion as

urn:oid:1.3.6.1.4.1.5923.1.1.1.3. However, it is optional to use this attribute.

For example:

Bob@emc.com urn:ecs:iam::s3:saml-provider/provider1,urn:ecs:iam::s3:role/ ADFS-Dev urn:ecs:iam::s3:saml-provider/provider1,urn:ecs:iam::s3:role/ ADFS-Production ECS

User-specific access using SAML keys

Dell EMC recommends that you specify permissions based on the users identity when creating access policies in IAM.

As to create policies that contain user-specific information, the user identity should be available in SAML keys. The following SAML keys can be used in policy conditions to create unique user identifiers.

SAML keys Description

saml:namequalifier A hash value based on the concatenation of the Issuer response value (saml:iss) and a string with the ObjectScale account (account ID) and the friendly name (the last part of the ARN) of the SAML provider in IAM. The account (account ID) and provider name must be separated by a '/' as in "123456789012/provider_name".

The combination of NameQualifier and Subject can be used to uniquely identify a federated user. The following pseudocode shows how this value is calculated. In this pseudocode, "+" indicates concatenation, SHA1 represents a function that produces a message digest using SHA-1, and Base64 represents a function that produces Base-64 encoded version of the hash output.

Base64 = ( SHA1 ( "https://example.com/saml" + "ObjectScaleAccount" + "/SamlProvider" ) )

saml:sub This is the subject of the claim, which includes a value that uniquely identifies an individual user within an organization. For example, _3e52ef03414f3464d2461c00ebae0152c25fb88bbc.

saml:sub_type This key can be persistent, transient, or the full Format URI from the Subject and NameID elements used in your SAML assertion. A value of persistent indicates that the value in saml:sub is the same for a user across all sessions. If the value is transient, the user has a different saml:sub value for each session.

128 Accessing data with IAM and S3

IAM Policy

The following example shows a permission policy that uses the preceding keys to grant permissions to a user-specific folder in Amazon S3. The policy assumes that the Amazon S3 objects are identified using a prefix that includes both saml:namequalifier and saml:sub. Notice that the Condition element includes a test to be sure that saml:sub_type is set to persistent. If it is set to transient, the saml:sub value for the user can be different for each session, and the combination of values should not be used to identify user-specific folders.

{ "Version": "2012-10-17", "Statement": { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::exampleECSBucket/backup/${saml:namequalifier}/${saml:sub}", "arn:aws:s3:::exampleECSBucket/backup/${saml:namequalifier}/${saml:sub}/*" ], "Condition": {"StringEquals": {"saml:sub_type": "persistent"}} } }

Example with sample values

Create a role using AssumeRoleWithSAML.

Attach an IAM policy to this role as below.

{ "Version": "2012-10-17", "Statement": { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::exampleObjectScleBucket/backup/${saml:namequalifier}/${saml:sub}", "arn:aws:s3:::exampleObjectScle/backup/${saml:namequalifier}/${saml:sub}/*" ], "Condition": {"StringEquals": {"saml:sub_type": "persistent"}} } }

The values in the above example are as follows:

saml:iss = http://AD.adfs.emc.com/adfs/services/trust. See IAM supported condition keys for the list of SAML condition keys.

account = s3 providername = provider1 saml:sub = ADFS\Bob Base64 = SHA1 ("http://AD.adfs.emc.com/adfs/services/trust " + "s3" + "/provider1") SHA1 = BB9445BB2D9C57D519ACEBD08EFD428076522D5B Base64 of BB9445BB2D9C57D519ACEBD08EFD428076522D5B is u5RFuy2cV9UZrOvQjv1CgHZSLVs=.

Accessing data with IAM and S3 129

IAM Resource ARNs

Resource ARN formats and Unique ID Prefixes that are supported by ObjectScale IAM are described here.

Table 31. Global Replicated Entities

Entity ARN Unique ID Prefix

User urn:osc:iam: :user

OIDA

Group urn:osc:iam: :group

OGPA

Role urn:osc:iam: :role

OROA

Customer-managed policy urn:osc:iam: :policy

ONPA, ONVA(versioned policy)

ObjectScale system-managed policy

urn:osc:iam::policy ONPA

SAML provider urn:osc:iam: :saml-provider/

-

Federated user urn:osc:sts: :federated-user/

-

Active session with assume role urn:osc:sts: :assumed- role /

-

Access key - OKIA Temporary (STS) keys - OSI

Table 32. Bucket resources

Resource ARN Scope Operations

ObjectScale Bucket arn:aws:s3: : :

ObjectScale CRUD, List

Amazon S3 API support in ObjectScale ObjectScale supports the Amazon Simple Storage Service (Amazon S3) Application Programming Interface (API).

Table 33. S3 Object Service

Protocol Ports

HTTP 9020

HTTPS 9021

S3 API support in ObjectScale

S3 API are supported in this release of ObjectScale.

S3 API

Refer to the ObjectScale Rest API zip file on Dell EMC Support at https://www.dell.com/support/home/en-us/product- support/product/objectscale/docs for a complete list of the supported S3 APIs.

130 Accessing data with IAM and S3

S3 Select

Additionally, in this release ObjectScale supports Amazon S3 Select. Refer to https://docs.aws.amazon.com/AmazonS3/latest/ userguide/s3-glacier-select-sql-reference-select.html for details on the Amazon S3 Select usages supported on ObjectScale.

The S3 Select API allows applications to retrieve a subset of an object's data by using SQL expressions. This can save on both bandwidth and processing. Using an example 2GB .csv object, without S3 Select the application would have to download the entire 2GB object and then do the processing on that data. With S3 Select, the application instead issues SQL select command and gets potentially only a small subset of that data. Further, the application does not have to do additional processing.

S3 Select can be used for objects in following formats:

.csv .json .parquet S3 Select supports querying .gzip and .bzip2 compressed objects of the above file types.

Though S3 Select API can be used on it's own it is very common for it to be used by query engines, like presto. A connector in presto can determine if a particular query can be sent directly to the storage, for example s3 select pushdown.

ObjectScale S3 error codes

The error codes that can be generated by the ObjectScale S3 are listed in the following table.

Table 34. Error Codes

Error Code HTTP Status Code

Generic Error Code Description Error

AccessDenied 403 AccessDenied Access Denied

BadDigest 400 BadDigest The Content-MD5 you specified did not match that received.

BucketAlreadyExists 409 BucketAlreadyExists The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.

BucketNotEmpty 409 BucketNotEmpty The bucket you tried to delete is not empty.

ContentMD5Empty 400 InvalidDigest The Content-MD5 you specified was invalid.

ContentMD5Missing 400 InvalidRequest The required Content-MD5 header for this request is missing.

EntityTooSmall 400 EntityTooSmall The proposed upload is smaller than the minimum allowed object size.

EntityTooLarge 400 EntityTooLarge The proposed upload exceeds the maximum allowed object size.

IncompleteBody 400 IncompleteBody The number of bytes specified by the Content-Length HTTP header were not provided.

InternalError 500 InternalError An internal error was encountered. Please try again.

ServerTimeout 500 ServerTimeout An internal timeout error was encountered. Please try again.

InvalidAccessKeyId 403 InvalidAccessKeyId The Access Key Id you provided does not exist.

Accessing data with IAM and S3 131

Table 34. Error Codes (continued)

Error Code HTTP Status Code

Generic Error Code Description Error

InvalidArgument 400 InvalidArgument Invalid Argument.

NoNamespaceForAnonymousRequest 403 AccessDenied ObjectScale could not determine the namespace from the anonymous request. Please use a namespace BaseURL or include an x-emc- namespace header.

InvalidBucketName 400 InvalidBucketName The specified bucket is not valid.

InvalidDigestBadMD5 400 InvalidDigest The Content-MD5 you specified was invalid.

InvalidDigest 403 SignatureDoesNotMatch The Content-MD5 you specified was an invalid.

InvalidRequest 400 InvalidRequest Invalid Request.

InvalidPart 400 InvalidPart One or more of the specified parts could not be found. The part might not have been uploaded.

InvalidPartOrder 400 InvalidPartOrder The list of parts was not in ascending order. Parts list must specified in order by part number.

InvalidPartSizeZero 400 InvalidPartSizeZero The upload part size cannot be zero.

MissingEncryption 400 InvalidRequest The multipart upload initiate requested encryption. Subsequent part requests must include the appropriate encryption parameters.

NoEncryptionNeed 400 InvalidRequest The multipart initiate request did not request encryption. Please resend the request without sending encryption parameters.

BadMD5 400 InvalidRequest The calculated MD5 hash of the key did not match the hash that was provided.

BadEncryptKey 400 InvalidRequest The provided encryption parameters did not match the ones used originally.

InvalidRange 416 InvalidRange The requested range cannot be satisfied.

KeyTooLong 400 KeyTooLong The specified key is too long.

MalformedACLError 400 MalformedACLError The XML provided was not well- formed or did not validate against the ObjectScale published schema.

MalformedXML 400 MalformedXML Malformed xml (that does not conform to the published xsd) for the configuration was sent.

MaxMessageLengthExceeded 400 MaxMessageLengthExceeded The request was too big.

MetadataTooLarge 400 MetadataTooLarge The metadata headers exceed the maximum allowed metadata size. *

InvalidProject 400 InvalidProject The specified project is Invalid.

132 Accessing data with IAM and S3

Table 34. Error Codes (continued)

Error Code HTTP Status Code

Generic Error Code Description Error

InvalidVPool 400 InvalidVPool The specified vPool (Replication Group) is Invalid.

InvalidNamespace 400 InvalidNamespace The specified namespace is Invalid.

MethodNotAllowed 405 MethodNotAllowed The specified method is not allowed against this resource.

MissingContentLength 411 MissingContentLength The Content-Length HTTP header must be provided.

MissingRequestBodyError 400 MissingRequestBodyError An empty XML document was sent. The error message is: Request body is empty.

MissingSecurityHeader 400 MissingSecurityHeader The equest was missing a required header.

IncompleteLifecycleConfig 400 IncompleteLifecycleConfig At least one action needs to be specified in a rule.

MalformedLifecycleConfig 400 MalformedLifecycleConfig The XML provided was not well- formed or did not validate against the published schema.

MalformedDateLifecycleConfig 400 MalformedDateLifecycleConfig The XML provided was not well- formed or did not validate against the published schema. Invalid Date or Days.

NoSuchBucket 404 NoSuchBucket The specified bucket does not exist.

NoSuchBucketPolicy 404 NoSuchBucketPolicy The bucket policy does not exist.

NoSuchKey 404 NoSuchKey The specified key does not exist.

NoSuchRetention 404 NoSuchRetention The specified retention does not exist.

ObjectUnderRetention 409 ObjectUnderRetention The object is under retention and cannot be deleted or modified.

NoSuchUpload 404 NoSuchUpload The specified multipart upload does not exist. The upload ID might be invalid.

NotImplemented 501 NotImplemented The requested functionality is not implemented.

OperationAborted 409 OperationAborted A conflicting conditional operation is currently in progress against this resource. Please try again.

PermanentRedirect 301 PermanentRedirect The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

PreconditionFailed 412 PreconditionFailed At least one of the preconditions you specified did not hold.

RequestIsNotMultiPartContent 400 RequestIsNotMultiPartContent Bucket POST must be of the enclosure type multipart/form- data.

Accessing data with IAM and S3 133

Table 34. Error Codes (continued)

Error Code HTTP Status Code

Generic Error Code Description Error

RequestTimeout 400 RequestTimeout The socket connection to the server was not read from or written to within the timeout period.

RequestTimeTooSkewed 403 RequestTimeTooSkewed The difference between the request time and the server's time is too large.

DateIsRequired 403 AccessDenied A valid Date or x-amz-date header is required.

SignatureDoesNotMatch 403 SignatureDoesNotMatch The request signature calculated does not match the signature provided. Check the Secret Access Key and signing method.

ZeroAmzExpires 403 Forbidden Zero value specified for x-amz- expires.

InvalidAmzExpires 400 Bad Request Invalid value specified for x-amz- expires.

ServiceUnavailable 503 ServiceUnavailable Please reduce your request rate.

TemporaryRedirect 307 TemporaryRedirect Requests are being redirected to the bucket while DNS updates.

TooManyBuckets 400 TooManyBuckets The request attempted to create more buckets than allowed.

UnexpectedContent 400 UnexpectedContent The request does not support this content.

UnresolvableGrantByEmailAddress 400 UnresolvableGrantByEmailAddress The email address you provided does not match any account on record.

InvalidBucketState 409 InvalidBucketState The request is not valid with the current state of the bucket.

SlowDown 503 SlowDown Please reduce your request rate.

AccountProblem 403 AccountProblem There is a problem with the specified account that prevents the operation from completing successfully.

CrossLocationLoggingProhibited 403 CrossLocationLoggingProhibited Cross location logging is not allowed. Buckets in one geographic location cannot log information to a bucket in another location.

ExpiredToken 400 ExpiredToken The provided token has expired.

IllegalVersioningConfigurationExcepti on

400 IllegalVersioningConfigurationExcepti on

The Versioning configuration specified in the request is invalid.

IncorrectNumberOfFilesInPostReques t

400 IncorrectNumberOfFilesInPostReques t

POST requires exactly one file upload per request.

InvalidAddressingHeader 500 InvalidAddressingHeader The specified role must be Anonymous role.

InvalidLocationConstraint 400 InvalidLocationConstraint The specified location constraint is not valid.

134 Accessing data with IAM and S3

Table 34. Error Codes (continued)

Error Code HTTP Status Code

Generic Error Code Description Error

InvalidPolicyDocument 400 InvalidPolicyDocument The content of the form does not meet the conditions specified in the policy document.

InvalidStorageClass 400 InvalidStorageClass The storage class you specified is not valid.

InvalidTargetBucketForLogging 400 InvalidTargetBucketForLogging The target bucket for logging does not exist, is not owned by you, or does not have the appropriate grants for the log delivery group.

InvalidToken 400 InvalidToken The provided token is malformed or otherwise invalid.

InvalidURI 400 InvalidURI Unable to parse the specified URI.

MalformedPOSTRequest 400 MalformedPOSTRequest The body of the POST request is not well-formed multipart/form- data.

MaxPostPreDataLengthExceededErr or

400 MaxPostPreDataLengthExceededErr or

The POST request fields preceding the upload file were too large.

NoLoggingStatusForKey 400 NoLoggingStatusForKey There is no such thing as a logging status subresource for a key.

NoSuchLifecycleConfiguration 404 NoSuchLifecycleConfiguration The lifecycle configuration does not exist.

NoSuchVersion 404 NoSuchVersion Indicates that the version ID specified in the request does not match an existing version.

RequestTorrentOfBucketError 400 RequestTorrentOfBucketError Requesting the torrent file of a bucket is not permitted.

UserKeyMustBeSpecified 400 UserKeyMustBeSpecified The bucket POST must contain the specified field name. If it is specified please check the order of the fields.

AmbiguousGrantByEmailAddress 400 AmbiguousGrantByEmailAddress The email address you provided is associated with more than one account.

BucketAlreadyOwnedByYou 409 BucketAlreadyOwnedByYou The previous request to create the named bucket succeeded and you already own it.

CredentialsNotSupported 400 CredentialsNotSupported The request does not support credentials.

InlineDataTooLarge 400 InlineDataTooLarge The inline data exceeds the maximum allowed size.

InvalidPayer 403 InvalidPayer All access to this object has been disabled.

TokenRefreshRequired 400 TokenRefreshRequired The provided token must be refreshed.

AccessModeNotSupported 409 AccessModeNotSupported The bucket does not support file access or the requested access mode is not allowed.

Accessing data with IAM and S3 135

Table 34. Error Codes (continued)

Error Code HTTP Status Code

Generic Error Code Description Error

AccessModeInvalidToken 409 AccessModeInvalidToken The token for the file access switch request is invalid.

NoSuchBaseUrl 400 NoSuchBaseUrl The specified BaseUrl does not exist.

NoDataStoreForVirtualPool 404 NoDataStoreForVirtualPool No Data Store found for Replication Group of the bucket.

VpoolAccessNotAllowed 400 Cannot AccessVpool Bucket is hosted on a Replication Group that is not accessible from S3.

InvalidCorsRequest 403 InvalidCorsRequest Invalid CORS request.

InvalidCorsRule 400 InvalidCorsRule Invalid CORS rule.

NoSuchCORSConfiguration 404 NoSuchCORSConfiguration The CORS configuration does not exist.

InvalidAclRequest 404 NoACLFound The ACL does not exist.

InsufficientStorage 507 InsufficientStorage The server cannot process the request because there is not enough space on disk.

BadMaxParts 400 InvalidArgument Argument max-parts must be an integer between 0 and 2147483647.

BucketNotFound 404 NoSuchBucket The specified bucket does not exist.

NotSupported 400 Not Supported The bucket may be locked.

InvalidContentLength 400 Invalid content length The content length has invalid value.

InvalidVersioningRequest 403 Invalid request for version control The bucket is in compliance mode.

InvalidLifeCycleRequest 403 Invalid request for life cycle The bucket is in compliance mode.

RetentionPeriodRequired 400 Invalid request for bucket with compliance

The bucket requires a retention period.

Conflict 409 Conflict The bucket may be locked.

MethodForbidden 403 Forbidden Check if quota has been exceeded.

NotAcceptable 406 Content encoding not acceptable The object Content-Encoding does not match requested Accept- Content.

InvalidEncoding 400 Invalid URL enconding The URL encoding used is invalid.

InvalidMetadataQuery 400 Invalid metadata query entered The metadata query entered does not conform to valid syntax

InvalidMetadataSearchList 400 Invalid metadata search list entered A keyname on the request is not a valid indexable key, or the format of the request list is incorrect.

MetadataSearchNotEnabled 405 Metadata search not enabled Metadata search is not enabled for this bucket.

MetadataSearchBadParameter 400 Metadata search invalid parameter used in query

Invalid search index key name, sort key name or attribute name value.

MetadataSearchInvalidArgument 400 Metadata search invalid parameter used in query

Invalid search index value format or operator used.

136 Accessing data with IAM and S3

Table 34. Error Codes (continued)

Error Code HTTP Status Code

Generic Error Code Description Error

MetadataSearchInvalidValuefor Datatype

400 Metadata search key indexing found invalid input value

Object operation failed because a user metadata value cannot be converted to its defined datatype.

MetadataOperationNotSupported 405 Metadata search operation not supported

Metadata query with both AND and OR logical operators not supported.

MetadataSearchBadSortParameter 400 Metadata search invalid sort parameter

The sort parameter has to be present in the query as a search parameter.

MetadataSearchRestriction 400 Buckets that are encrypted or within an encrypted namespace cannot have metadata search enabled

Metadata search is mutually exclusive with bucket/namespace encryption.

MetadataSearchTooManyIndexKeys 400 The number of Index keys exceeds the maximum allowed

The number of keys to be indexed exceeds the maximum number allowed, try with fewer keys.

InvalidOrNoCustomerProvided EncryptionKey

400 Invalid or no customer provided encryption key

No encryption key, or an encryption key that did not match the one in the system, was provided.

DareUnavailable 403 Server side encryption (D@RE) is not supported

D@RE JAR/license is unavailable hence server side encryption requests are not supported.

SelfCopyInvalidRequest 400 InvalidRequest The copy request is illegal because it is trying to copy an object to itself without changing the object's metadata or encryption attributes.

OverLappingPrefixes 400 Invalid Request Found overlapping prefixes.

SamePrefix 400 Invalid Request Found two rules with same prefix.

XAmzContentSHA256Mismatch 400 XAmzContentSHA256Mismatch The Content-SHA256 you specified did not match what we received

InvalidJSON 400 InvalidJSON Policies must be valid JSON and the first byte must be {.

InvalidBucketPolicy 400 InvalidBucketPolicy Invalid Bucket Policy.

MalformedPolicy 400 MalformedPolicy Malformed Policy.

MaxIDLengthExceeded 400 InvalidArgument ID length should not exceed allowed limit of 255.

CrossHeadAccessBeforeUpgrade 400 InvalidRequest Cross head access is not supported.

InvalidDate 400 InvalidArgument Date must be no earlier than 1970-01-01T00:00:00.000Z.

BadContentLengthRequest 400 RequestTimeout Content-Length specified is not matching with Length of the Content in the body.

IncompatibleNode 500 InternalError The I/O request was sent to the wrong node, please configure client or Load-balancer correctly to route request to the correct node.

InvalidFileNameArgument 400 InvalidArgument Header value cannot be represented using ISO-8859-1.

Accessing data with IAM and S3 137

Table 34. Error Codes (continued)

Error Code HTTP Status Code

Generic Error Code Description Error

InvalidPartNumber 400 InvalidPartNumber Part number must be an integer between 1 and 10000, inclusive

InvalidTenant 400 TenantNotFound Specified Tenant is Invalid.

Redirect 307 Redirect Temporary redirect.

UrlAmzExpires 403 Forbidden Request has expired

VersioningCannotChange 409 InvalidBucketState An Object Lock configuration is present on this bucket, so the versioning state cannot be changed.

ObjectLockNotEnabled 409 InvalidBucketState Object Lock configuration cannot be enabled on existing buckets

ObjectLockConfigurationNotFoundEr ror

404 ObjectLockConfigurationNotFoundEr ror

Object Lock configuration does not exist for this bucket

ObjectLockConfigMalformedXML 400 MalformedXML The XML you provided was not well- formed or did not validate against our published schema.

ObjectLockConfigInvalidArgument 400 InvalidArgument Default retention period must be a positive integer value.

ObjectLockConfigInvalidArgumentTo oLarge

400 InvalidArgument Default retention period too large.

ObjectLockMalformedXML 400 MalformedXML The XML you provided was not well- formed or did not validate against our published schema.

ObjectLockConfigurationMissing 400 InvalidRequest Bucket is missing Object Lock Configuration

ObjectLockNoSuchObjectLockConfig uration

404 NoSuchObjectLockConfiguration The specified object does not have a ObjectLock configuration

ObjectLockRetailUntilMustBeInFuture 400 InvalidArgument The retain until date must be in the future!

ObjectLockAccessDenied 403 AccessDenied Access Denied

ObjectLockAndAdo 400 InvalidRequest Object Lock enabled bucket is not compatible with ADO.

ObjectLockAndFsa 400 InvalidRequest Object Lock enabled bucket is not compatible with File System Access.

ObjectLockAccessDeniedNonIAM 403 AccessDenied Only IAM users are supported with object lock enabled buckets.

ObjectLockMissingHeader 400 InvalidArgument x-amz-object-lock-retain-until-date and x-amz-object-lock-mode must both be supplied

ObjectLockUnknownModeDirective 400 InvalidArgument Unknown Mode directive.

ObjectLockBadDateFormat 400 InvalidArgument The retain until date must be provided in ISO 8601 format

InvalidVersionId 400 InvalidArgument Invalid version id specified.

AccessModeInvalidToken 409 AccessModeInvalidToken The token for the file access switch request is invalid

138 Accessing data with IAM and S3

Table 34. Error Codes (continued)

Error Code HTTP Status Code

Generic Error Code Description Error

UnmodifiedSince 304 Not modified it has not been modified since the specified time

Match 304 Not modified its entity tags (ETag) are not different from the one specified

ObjectRetentionPeriodRequired 400 RetentionPeriodRequired The retention period value is required

ObjectRetentionCannotBeDecreased 400 RetentionCannotBeDecreased The new retention period value must be greater than current

MetadataSearchInvalidQueryMarker 400 InvalidArgument The marker provided is incorrect

InvalidMetadataSearchKeys 400 Invalid metadata key Duplicate metadata key with different type in search list entered

NoContent 204 NoContent

INVALID_CONTINUATION_TOKEN 400 InvalidArgument The continuation token provided is incorrect

ContentTypeMissing 400 InvalidRequest Missing required header for this request:Content-Type

ContentTypeArgMissing 400 InvalidArgument Content-Type missing for object in CopyRangeRequest

CopyModeMissing 400 Bad Request Invalid x-emc-copy-mode value

InvalidCopyPath 400 InvalidCopySource The path of source segment is invalid or not found

InvalidETag 400 InvalidArgument The etag of source segment miss matching

EmptyRequestBody 400 InvalidRequest Empty request body is not allow for copy range API

Invalid_Copy_Range 400 InvalidCopyRange Invalid source object range provide

ACCESS_DENIED_SOURCE_OBJEC T

400 InvalidArgument Access denied reading one or more source objects

Max_Copy_Ranges_Exceed 400 MaxMessageLengthExceeded Your request exceeded the maximum number of segments (250).

Invalid_LifeCycle_Version_Config 400 InvalidRequest can't support mixed version lifecycle config

Metadata_Not_Allowed 400 InvalidArgument Metadata cannot be specified in this context

Invalid_Part_Number 416 InvalidPartNumber The requested partnumber is not satisfiable

UnSupportedV2ListingParams 400 InvalidArgument Unsupported query parameter with GET.BUCKET in list-type=2

UnSupportedV1ListingParams 400 InvalidArgument One or more query parameters only supported in GET.BUCKET with list- type=2

InvalidArgumentVersion 400 InvalidArgument Version is not supported in this request

InvalidLifecycleDays 400 InvalidArgument Days for lifecycle action must be a positive integer

Accessing data with IAM and S3 139

Table 34. Error Codes (continued)

Error Code HTTP Status Code

Generic Error Code Description Error

InvalidLifecycleRuleId 400 InvalidArgument Rule ID must be unique. Found same ID for more than one rule

Invalid_Index_Granularity 400 InvalidIndexGranularity Invalid value specified for x-emc- index-granularity

MaxLifecycleRulesLimitExceed 400 MalformedXML Number of rules should not exceed allowed limit of 1000

MisMatchDare 400 InvalidRequest Missing Encryption parameters or the one provided does not match the original.

MetadataPrefixSearchBadParameter 400 invalid parameter used in query query on ObjectName not supported with prefix

UnSupportedCopyRangeRequest 403 Forbidden IAM user is not supported for Copy Range API

CastFailed 400 CastFailed Attempt to convert from one data type to another using CAST failed in the SQL expression.

ColumnTooLong 400 ColumnTooLong The length of a column in the result is greater than maxCharsPerColumn of 1 MB.

CSVEscapingRecordDelimiter 400 CSVEscapingRecordDelimiter Quoted record delimiter found in the file. To allow quoted record delimiters, please set AllowQuotedRecordDelimiter to 'TRUE'.

CSVParsingError 400 CSVParsingError Encountered an error parsing the CSV file. Check the file and try again.

CSVUnescapedQuote 400 CSVUnescapedQuote Unescaped quote found while parsing the .csv file. Ensure that AllowQuotedRecordDelimiter is set to 'TRUE' if quoted record delimiters are present.

ExpressionTooLong 400 ExpressionTooLong The SQL expression is too long: The maximum byte-length for the SQL expression is 256 KB.

EvaluatorBindingDoesNotExist 400 EvaluatorBindingDoesNotExist A column name or a path provided does not exist in the SQL expression.

EvaluatorInvalidArguments 400 EvaluatorInvalidArguments Incorrect number of arguments in the function call in the SQL expression.

EvaluatorInvalidTimestampFormatPat tern

400 EvaluatorInvalidTimestampFormatPat tern

Invalid timestamp format string in the SQL expression.

EvaluatorTimestampFormatPatternD uplicateFields

400 EvaluatorTimestampFormatPatternD uplicateFields

Timestamp format pattern contains multiple format specifiers representing the timestamp field in the SQL expression.

EvaluatorTimestampFormatPatternH ourClockAmPmMismatch

400 EvaluatorTimestampFormatPatternH ourClockAmPmMismatch

Timestamp format pattern contains a 12-hour hour of day format symbol but doesn't also contain an AM/PM field, or it contains a 24-hour hour of

140 Accessing data with IAM and S3

Table 34. Error Codes (continued)

Error Code HTTP Status Code

Generic Error Code Description Error

day format specifier and contains an AM/PM field in the SQL expression.

EvaluatorInvalidTimestampFormatPat ternSymbolForParsing

400 EvaluatorInvalidTimestampFormatPat ternSymbolForParsing

Timestamp format pattern contains a valid format symbol that cannot be applied to timestamp parsing in the SQL expression.

EvaluatorUnterminatedTimestampFor matPatternToken

400 EvaluatorUnterminatedTimestampFor matPatternToken

Timestamp format pattern contains unterminated token in the SQL expression.

EvaluatorInvalidTimestampFormatPat ternToken

400 EvaluatorInvalidTimestampFormatPat ternToken

Timestamp format pattern contains an invalid token in the SQL expression.

EvaluatorInvalidTimestampFormatPat ternSymbol

400 EvaluatorInvalidTimestampFormatPat ternSymbol

Timestamp format pattern contains an invalid symbol in the SQL expression.

IllegalSqlFunctionArgument 400 IllegalSqlFunctionArgument Illegal argument was used in the SQL function.

InvalidColumnIndex 400 InvalidColumnIndex Column index in the SQL expression is invalid.

InvalidCompressionFormat 400 InvalidCompressionFormat The file is not in a supported compression format. Only GZIP and BZIP2 are supported.

InvalidExpressionType 400 InvalidExpressionType The ExpressionType is invalid. Only SQL expressions are supported.

InvalidFileHeaderInfo 400 InvalidFileHeaderInfo The FileHeaderInfo is invalid. Only NONE, USE, and IGNORE are supported.

InvalidKeyPath 400 InvalidKeyPath Key path in the SQL expression is invalid.

InvalidJsonType 400 InvalidJsonType The JsonType is invalid. Only DOCUMENT and LINES are supported.

InvalidQuoteFields 400 InvalidQuoteFields The QuoteFields is invalid. Only ALWAYS and ASNEEDED are supported.

InvalidRequestParameter 400 InvalidRequestParameter The value of a parameter in SelectRequest element is invalid. Check the service API documentation and try again.

OverMaxColumn 400 OverMaxColumn The number of columns in the result is greater than the maximum allowable number of columns.

OverMaxRecordSize 400 OverMaxRecordSize The length of a record in the input or result is greater than maxCharsPerRecord of 1 MB.

TruncatedInput 400 TruncatedInput Object decompression failed. Check that the object is properly compressed using the format specified in the request.

Accessing data with IAM and S3 141

Table 34. Error Codes (continued)

Error Code HTTP Status Code

Generic Error Code Description Error

UnauthorizedAccess 401 UnauthorizedAccess You are not authorized to perform this operation.

ExternalEvalException 400 ExternalEvalException The query cannot be evaluated. Check the file and try again.

InvalidDataSource 400 InvalidDataSource Invalid data source type. Only CSV, JSON, and Parquet are supported.

InvalidDataType 400 InvalidDataType The SQL expression contains an invalid data type.

InvalidTableAlias 400 InvalidTableAlias The SQL expression contains an invalid table alias.

InvalidTextEncoding 400 InvalidTextEncoding Invalid encoding type. Only UTF-8 encoding is supported.

JSONParsingError 400 JSONParsingError Encountered an error parsing the JSON file. Check the file and try again.

UnrecognizedFormatException 400 UnrecognizedFormatException Encountered an invalid record type.

MissingRequiredParameter 400 MissingRequiredParameter The SelectRequest entity is missing a required parameter. Check the service documentation and try again.

S3SelectNoMemory 503 S3SelectNoMemory Not enough memory available for the SelectRequest.

MultipleDataSourcesUnsupported 400 MultipleDataSourcesUnsupported Multiple data sources are not supported.

ObjectSerializationConflict 400 ObjectSerializationConflict InputSerialization specifies more than one format (CSV, JSON, or Parquet), or OutputSerialization specifies more than one format (CSV or JSON). InputSerialization and OutputSerialization can only specify one format each.

UnsupportedFunction 400 UnsupportedFunction Encountered an unsupported SQL function.

UnsupportedSqlOperation 400 UnsupportedSqlOperation Encountered an unsupported SQL operation.

UnsupportedSqlStructure 400 UnsupportedSqlStructure Encountered an unsupported SQL structure. Check the SQL Reference.

UnsupportedStorageClass 400 UnsupportedStorageClass Encountered an invalid storage class. Only STANDARD, STANDARD_IA, and ONEZONE_IA storage classes are supported.

UnsupportedSyntax 400 UnsupportedSyntax Encountered invalid syntax.

UnsupportedRangeHeader 400 UnsupportedRangeHeader Range header is not supported for this operation.

LexerInvalidChar 400 LexerInvalidChar The SQL expression contains an invalid character.

LexerInvalidOperator 400 LexerInvalidOperator The SQL expression contains an invalid literal.

142 Accessing data with IAM and S3

Table 34. Error Codes (continued)

Error Code HTTP Status Code

Generic Error Code Description Error

LexerInvalidLiteral 400 LexerInvalidLiteral The SQL expression contains an invalid operator.

LexerInvalidIONLiteral 400 LexerInvalidIONLiteral The SQL expression contains an invalid operator.

ParseExpectedDatePart 400 ParseExpectedDatePart Did not find the expected date part in the SQL expression.

ParseExpectedKeyword 400 ParseExpectedKeyword Did not find the expected keyword in the SQL expression.

ParseExpectedTokenType 400 ParseExpectedTokenType Did not find the expected token in the SQL expression.

ParseExpected2TokenTypes 400 ParseExpected2TokenTypes Did not find the expected token in the SQL expression.

ParseExpectedNumber 400 ParseExpectedNumber Did not find the expected number in the SQL expression.

ParseExpectedRightParenBuiltinFunc tionCall

400 ParseExpectedRightParenBuiltinFunc tionCall

Did not find the expected right parenthesis character in the SQL expression.

ParseExpectedTypeName 400 ParseExpectedTypeName Did not find the expected type name in the SQL expression.

ParseExpectedWhenClause 400 ParseExpectedWhenClause Did not find the expected WHEN clause in the SQL expression. CASE is not supported.

ParseUnsupportedToken 400 ParseUnsupportedToken The SQL expression contains an unsupported token.

ParseUnsupportedLiteralsGroupBy 400 ParseUnsupportedLiteralsGroupBy The SQL expression contains an unsupported use of GROUP BY.

ParseExpectedMember 400 ParseExpectedMember The SQL expression contains an unsupported use of MEMBER.

ParseUnsupportedSelect 400 ParseUnsupportedSelect The SQL expression contains an unsupported use of SELECT.

ParseUnsupportedCase 400 ParseUnsupportedCase The SQL expression contains an unsupported use of CASE.

ParseUnsupportedCaseClause 400 ParseUnsupportedCaseClause The SQL expression contains an unsupported use of CASE.

ParseUnsupportedAlias 400 ParseUnsupportedAlias The SQL expression contains an unsupported use of ALIAS.

ParseUnsupportedSyntax 400 ParseUnsupportedSyntax The SQL expression contains unsupported syntax.

ParseUnknownOperator 400 ParseUnknownOperator The SQL expression contains an invalid operator.

ParseInvalidPathComponent 400 ParseInvalidPathComponent The SQL expression contains an invalid path component.

ParseMissingIdentAfterAt 400 ParseMissingIdentAfterAt Did not find the expected identifier after the @ symbol in the SQL expression.

Accessing data with IAM and S3 143

Table 34. Error Codes (continued)

Error Code HTTP Status Code

Generic Error Code Description Error

ParseUnexpectedOperator 400 ParseUnexpectedOperator The SQL expression contains an unexpected operator.

ParseUnexpectedTerm 400 ParseUnexpectedTerm The SQL expression contains an unexpected term.

ParseUnexpectedToken 400 ParseUnexpectedToken The SQL expression contains an unexpected token.

ParseUnExpectedKeyword 400 ParseUnExpectedKeyword The SQL expression contains an unexpected keyword.

ParseExpectedExpression 400 ParseExpectedExpression Did not find the expected SQL expression.

ParseExpectedLeftParenAfterCast 400 ParseExpectedLeftParenAfterCast Did not find the expected left parenthesis after CAST in the SQL expression.

ParseExpectedLeftParenValueConstr uctor

400 ParseExpectedLeftParenValueConstr uctor

Did not find expected the left parenthesis in the SQL expression.

ParseExpectedLeftParenBuiltinFuncti onCall

400 ParseExpectedLeftParenBuiltinFuncti onCall

Did not find the expected left parenthesis in the SQL expression.

ParseExpectedArgumentDelimiter 400 ParseExpectedArgumentDelimiter Did not find the expected argument delimiter in the SQL expression.

ParseCastArity 400 ParseCastArity The SQL expression CAST has incorrect arity.

ParseInvalidTypeParam 400 ParseInvalidTypeParam The SQL expression contains an invalid parameter value.

ParseEmptySelect 400 ParseEmptySelect The SQL expression contains an empty SELECT.

ParseSelectMissingFrom 400 ParseSelectMissingFrom The SQL expression contains a missing FROM after SELECT list.

ParseExpectedIdentForGroupName 400 ParseExpectedIdentForGroupName GROUP is not supported in the SQL expression.

ParseExpectedIdentForAlias 400 ParseExpectedIdentForAlias Did not find the expected identifier for the alias in the SQL expression.

ParseUnsupportedCallWithStar 400 ParseUnsupportedCallWithStar Only COUNT with (*) as a parameter is supported in the SQL expression.

ParseNonUnaryAgregateFunctionCall 400 ParseNonUnaryAgregateFunctionCall Only one argument is supported for aggregate functions in the SQL expression.

ParseMalformedJoin 400 ParseMalformedJoin JOIN is not supported in the SQL expression.

ParseExpectedIdentForAt 400 ParseExpectedIdentForAt Did not find the expected identifier for AT name in the SQL expression.

ParseAsteriskIsNotAloneInSelectList 400 ParseAsteriskIsNotAloneInSelectList Other expressions are not allowed in the SELECT list when '*' is used without dot notation in the SQL expression.

144 Accessing data with IAM and S3

Table 34. Error Codes (continued)

Error Code HTTP Status Code

Generic Error Code Description Error

ParseCannotMixSqbAndWildcardInSel ectList

400 ParseCannotMixSqbAndWildcardInSel ectList

Cannot mix [] and * in the same expression in a SELECT list in SQL expression.

ParseInvalidContextForWildcardInSel ectList

400 ParseInvalidContextForWildcardInSel ectList

Invalid use of * in SELECT list in the SQL expression.

ValueParseFailure 400 ValueParseFailure Timestamp parse failure in the SQL expression.

IncorrectSqlFunctionArgumentType 400 IncorrectSqlFunctionArgumentType Incorrect type of arguments in function call in the SQL expression.

AmbiguousFieldName 400 AmbiguousFieldName Field name matches to multiple fields in the file. Check the SQL expression and the file, and try again.

MissingHeaderName 400 MissingHeaderName Some headers in the query are missing from the file.

IntegerOverflow 400 IntegerOverflow Integer overflow or underflow in the SQL expression.

LikeInvalidInputs 400 LikeInvalidInputs Invalid argument given to the LIKE clause in the SQL expression.

InvalidCast 400 InvalidCast Attempt to convert from one data type to another using CAST failed in the SQL expression.

ParquetNotEnabled 400 ParquetNotEnabled Functionality for parsing Parquet format is not enabled.

ParquetParsingError 400 ParquetParsingError Error parsing Parquet file. Please check the file and try again.

NumberFormatError 400 NumberFormatError Error parsing a number. This can be caused by under/over flow of integers.

EvaluatorLikePatternInvalidEscapeSe quence

400 EvaluatorLikePatternInvalidEscapeSe quence

Invalid argument given to LIKE expression.

EvaluatorNegativeLimit 400 EvaluatorNegativeLimit LIMIT must not be negative.

OverMaxParquetBlockSize 400 OverMaxParquetBlockSize Parquet file is above the max row group size.

UnsupportedParquetType 400 UnsupportedParquetType Unsupported Parquet type.

ParquetUnsupportedCompressionCod ec

400 ParquetUnsupportedCompressionCod ec

Unsupported Parquet compression codec.

UnsupportedScanRangeInput 400 UnsupportedScanRangeInput Scan range queries are not supported on this type of object.

ErrorWritingRow 400 ErrorWritingRow Cannot format output for your query. Please check the file and query, and try again

ReplicationConfigurationNotFoundErr or

404 ReplicationConfigurationNotFoundErr or

The replication configuration was not found.

ReplicationStatusNotFoundError 404 ReplicationStatusNotFoundError Detailed replication status not found.

S3SelectOptionNotYetImplemented 400 S3SelectOptionNotYetImplemented The option specified not yet implemented.

Accessing data with IAM and S3 145

Table 34. Error Codes (continued)

Error Code HTTP Status Code

Generic Error Code Description Error

RANGE_UPDATE_NOT_SUPPORT 400 UnsupportedFeature Range update is not supported in current release

BucketNotificationMalformedArn 400 InvalidArgument The ARN is not well formed

BucketNotificationInvalidTopic 400 InvalidArgument Unable to validate the following destination configurations

BucketNotificationIdMaxLengthExcee ded

400 InvalidArgument ID length exceeded allowed limit of 255

BucketNotificationDuplicateId 400 InvalidArgument Same ID used for multiple configurations. IDs must be unique.

BucketNotificationUnsupportedEvent 400 InvalidArgument The event is not supported for notifications

BucketNotificationFilterPrefixLimitEx ceeded

400 InvalidArgument Cannot specify more than one prefix rule in a filter.

NOTE:

The PUT request header is limited to 8 KB in size. Within the PUT request header, the user-defined metadata is limited

to 2 KB in size. User-defined metadata is a set of key-value pairs. The size of user-defined metadata is measured by

taking the sum of the number of bytes in each key and value plus four: a colon and space to separate the name and

value and two bytes for carriage return-linefeed.

When the system throws a 500 error, it allows the user to retry the request. In such cases, it is recommended to

use a backoff algorithm which waits progressively longer between retries for consecutive error responses. For more

information about guidance on 500 error rate response in ObjectScale, see https://dell.com/support/objectscale.

Authenticating with the S3 service

The ObjectScale S3 service enables authentication using Signature Version 4. This topic identifies any ObjectScale-specific aspects of the authentication process.

Amazon S3 uses an authorization header that must be present in all requests to identify the user and provide a signature for the request.

In order to create an authorization header, you need an AWS Access Key Id and a Secret Access Key. In ObjectScale, the AWS Access Key Id is shown in the table located at Account > > Secret Key for each user. The Access Key ID is listed in the table.

The following notes apply:

When users add or change the secret key, they should wait two minutes for ObjectScale to refresh with the new secret key before using the new secret key.

Authenticating using Signature V4

The Authorization header when using Signature V4 looks like this:

Authorization: AWS4-HMAC-SHA256 Credential=OKIA60819103813C1F40/20130524/us/s3/aws4_request, SignedHeaders=host;range;x-amz-date, Signature=fe5f80f77d5fa3beca038a248ff027d0445342fe2855ddc963176630326f1024

The Credential component comprises your Access Key Id followed by the Credential Scope. The Credential Scope comprises Date/Region/Service Name/Termination String. For ObjectScale, the Service Name is always s3 and the Region can be any string. When computing the signature, ObjectScale uses the Region string passed by the client.

Authentication using Signature V4 is described in:

146 Accessing data with IAM and S3

http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html , and http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html

An example of a PUT replication request using Signature V4 is provided below:

PUT https:// /testbucket?replication Authorization: AWS4-HMAC-SHA256 Credential=OKIA60819103813C1F40/20160726/us/s3/ aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=e75a150daa28a2b2f7ca24f6fd0e161cb58648a25121d3108f0af5c9451b09ce Content-MD5: x0ns_8TT8w5fB2woe72A== Host: 10.247.195.130:9021 x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date: 20160726T033659Z

Response:

200 OK Date: Tue, 26 Jan 2022 03:37:00 GMT Server: ViPR/1.0 x-amz-request-id: 0af7c382:156123ab861:4192:896 x-amz-id-2: 3e2b2280876d444d6c7215091692fb43b87d6ad95b970f48911d635729a8f7ff

Use SDKs to access the S3 service

When developing applications that talk to the ObjectScale S3 service, there are a number of SDKs that support your development activity.

The following topics describe the use of the Amazon S3 SDK and the use of the ObjectScale Java S3 client SDK.

The ObjectScale Java S3 SDK is available at: https://github.com/EMCECS/objectscale-s3-client-java

The ObjectScale Community provides information about the various clients that are available and provides guidance on their use: https://www.dell.com/community/ObjectScale/bd-p/ObjectScale.

Using the AWS SDK for Java

You can access ObjectScale object storage using the AWS SDK for Java.

By default the AmazonS3Client client object is coded to work directly against amazon.com. This section shows how to set up the AmazonS3Client to work against ObjectScale.

In order to create an instance of the AmazonS3Client object, you need to pass it credentials. This is achieved through creating an AWSCredentials object and passing it the AWS Access Key (your ObjectScale user name) and your generated secret key for ObjectScale.

The following code snippet shows how to set this up.

AmazonS3Client client = new AmazonS3Client(new BasicAWSCredentials(uid, secret));

By default the Amazon client attempts to contact Amazon WebServices. In order to override this behavior and contact ObjectScale you need to set a specific endpoint.

You can set the endpoint using the setEndpoint method. The protocol specified on the endpoint dictates whether the client should be directed at either be a load balancer in Kubernetes or a NodePort, which would be an IP and a random high-level port, depending on your network configuration.

NOTE: If you intend to use the HTTPS port, the JDK of your application must be set up to validate the ObjectScale

certificate successfully; otherwise the client will throw SSL verification errors and fail to connect.

In the snippet below, the client is being used to access ObjectScale over HTTP:

AmazonS3Client client = new AmazonS3Client(new BasicAWSCredentials(uid, secret)); client.setEndpoint("http:// : ");

Accessing data with IAM and S3 147

When using path-style addressing (objs1.dell.com/mybucket), you will need to set the setPathStyleAccess option, as shown below:

S3ClientOptions options = new S3ClientOptions(); options.setPathStyleAccess(true);

AmazonS3Client client = new AmazonS3Client(new BasicAWSCredentials(uid, secret)); client.setEndpoint("http:// : "); client.setS3ClientOptions(options);

The following code shows how to list objects in a bucket.

ObjectListing objects = client.listObjects("mybucket"); for (S3ObjectSummary summary : objects.getObjectSummaries()) { System.out.println(summary.getKey()+ " "+summary.getOwner()); }

The CreateBucket operation differs from other operations in that it expects a region to be specified. Against S3 this would indicate the data center in which the bucket should be created. However, ObjectScale does not support regions. For this reason, when calling the CreateBucket operation, we specify the standard region, which stops the AWS client from downloading the Amazon Region configuration file from Amazon CloudFront.

client.createBucket("mybucket", "Standard");

The complete example for communicating with the ObjectScale S3 data service, creating a bucket, and then manipulating an object is provided below:

public class Test { public static String uid = "root"; public static String secret = "KHBkaH0Xd7YKF43ZPFbWMBT9OP0vIcFAMkD/9dwj"; public static String s3Endpoint = "http:// : ";

public static String bucketName = "myBucket"; public static File objectFile = new File("/photos/cat1.jpg");

public static void main(String[] args) throws Exception {

AmazonS3Client client = new AmazonS3Client(new BasicAWSCredentials(uid, secret));

S3ClientOptions options = new S3ClientOptions(); options.setPathStyleAccess(true);

AmazonS3Client client = new AmazonS3Client(credentials); client.setEndpoint(s3Endpoint); client.setS3ClientOptions(options);

client.createBucket(bucketName, "Standard"); listObjects(client);

client.putObject(bucketName, objectFile.getName(), objectFile); listObjects(client);

client.copyObject(bucketName,objectFile.getName(),bucketName, "copy-" + objectFile.getName()); listObjects(client); }

public static void listObjects(AmazonS3Client client) { ObjectListing objects = client.listObjects(bucketName); for (S3ObjectSummary summary : objects.getObjectSummaries()) { System.out.println(summary.getKey()+ " "+summary.getOwner()); } } }

148 Accessing data with IAM and S3

ObjectScale Java S3 client SDK

The ObjectScale Java S3 client SDK is a library to assist users of the ObjectScale platform. This includes an API to interact with ObjectScale's own API..

Requirements:

Java 11 or higher ObjectScale 1.0.0 or higher.

An example of using this SDK (S3Client) is shown below for metadatasearch.

package main.java.metadatasearch;

import com.dellemc.objectscale.s3.ObjectScaleS3Client; import com.dellemc.objectscale.s3.model.*; import com.dellemc.objectscale.s3.ObjectScaleS3ClientBuilder;

import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; import software.amazon.awssdk.services.s3.S3Client; import software.amazon.awssdk.services.s3.model.Bucket; import software.amazon.awssdk.services.s3.model.ListBucketsRequest; import software.amazon.awssdk.services.s3.model.ListBucketsResponse; import software.amazon.awssdk.services.s3.model.S3Exception;

import java.net.URI;

// This is an example of a program which creates 10 buckets with // ObjectScale's metadata search enabled, and then queries all buckets in a store // for all objects created after Jan 1 2015. public abstract class SearchMetadata {

// You can adjust these according to your setup. static final String S3_IP = "127.0.0.1"; static final String S3_PORT = "80"; static final String ACCESS_KEY = "OKIA----------------"; static final String SECRET = "----------------------------------------"; static final String BUCKET = "bucket-metadata-search-example";

static ObjectScaleS3Client client;

// This is an example of how one can create buckets with ObjectScale's metadata search feature // enable, get a list of all buckets, and search for metadata using the query objects endpoint // with selectors. public static void main( String[] args ) { ObjectScaleS3ClientBuilder builder = ObjectScaleS3Client.builder() .endpointOverride(URI.create("http://"+S3_IP+":"+S3_PORT)) .credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials .create(ACCESS_KEY, SECRET))); client = builder.build();

// Create a set of buckets for ( int i = 1; i <= 10; i++ ) { createTestBuckets(client, BUCKET+"-"+i); }

// Get a list of the current buckets ListBucketsRequest listRequest = ListBucketsRequest.builder().build(); ListBucketsResponse listResponse = client.listBuckets(listRequest);

for ( Bucket b : listResponse.buckets() ){ System.out.println("===== Bucket " + b.name() + " ====="); try { // Query the objects in the bucket for anything modified after january 1st, 2015. QueryObjectsRequest qo = QueryObjectsRequest.builder().bucket(b.name()).query("LastModified>2015-01-01T00:00:00Z") .build();

Accessing data with IAM and S3 149

QueryObjectsResponse resp = client.queryObjects(qo); // For every Object... for ( QueryObject o : resp.objects() ) { // For every queried metadata set for ( QueryMetadata m : o.queryMetadata() ) { // For every key in the metadata map for ( String s : m.metadataMap().keySet() ) { // Print out the info. System.out.println(o.objectName() + ": " + m.typeAsString() + ": " + s + " " + m.metadataMap().get(s)); } } } } catch ( S3Exception e ){ if( e.getLocalizedMessage().startsWith("Invalid search index key name") ) {} System.out.println("metadata search not enabled on this bucket, or key not searchable"); } } }

// Create a bucket with a given name and client where one can query / filter based // on the LastModified field. public static void createTestBuckets( S3Client client, String name ){ CreateBucketRequest createBucketRequest = CreateBucketRequest.builder() .metadataSearchKeys("LastModified;datetime").bucket(name).build(); // Use toStandardRequest to use this as a CreateBucketRequest client.createBucket(createBucketRequest.toStandardRequest()); } }

Working with S3 workloads in ObjectScale

After setting up accounts and users as well as an object store and bucket, you can perform S3 workloads using the ObjectScale instance object storage.

Record S3 endpoint values

About this task

Use this table to record with the values used to create the bucket. These three S3 values are required for application access to the endpoint.

Object store name

Bucket name

ObjectScale Name S3 Browser Name Your Value

ACCESS KEY Access Key ID

SECRET_KEY Secret Access Key

EXTERNAL_ENDPOINT REST Endpoint

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Collect the S3 endpoint value from the object store Summary tab.

150 Accessing data with IAM and S3

Record this value in the EXTERNAL_ENDPOINT value in the table above.

4. Locate the Secret Key .csv file previously saved locally for the user that owns the bucket in the object store.

This user should be a part of the IAM account that is a tenant within the selected object store.

Record this value as the SECRET_KEY in the table above.

5. Finally, collect the Access Key ID for the user.

a. Go to the object store Accounts tab and click on the name of the IAM account that manages the user account. b. Select the Users tab and click on the name of the user account that will be used for S3. c. Click on Secret Key and record the Access Key ID value displayed in the Secret Key table.

NOTE: If you do not have the previously created Secret Key for this user or wish to change it for any reason, you

can DEACTIVATE or REMOVE old Secret Keys/Access Key pairs and click ADD KEY to generate a new key for the

user.

Verify S3 access

About this task

Use the freely available S3 Browser or similar application to verify connectivity to the object store owned by the initial-IAM user setup during deployment.

Steps

1. In S3 browser, create an account with the details of the object store bucket.

Use the S3 values you recorded in the last task to complete this step.

a. Type a Name for the account. b. Select S3 Compatible Storage from the Account Type dropdown. c. Enter the EXTERNAL_ENDPOINT value into the REST Endpoint field.

For example:

Rest Endpoint: 10.55.66.77:443

d. Enter the ACCESS KEY value into the Access Key ID field.

For example:

Access Key: AKIA5F587FA0E4E4FF81

e. Enter the SECRET_KEY value into the Secret Access Key field.

For example:

Secret Key: KqM5xHvaG7Bv9SH0lGoMrGYDWJrUoZsVvZ71JBeY

2. Connectivity is verified by creating new buckets and uploading objects via S3 Browser or a similar S3 Compatible application.

3. Optional: Create additional buckets using S3 browser or within the object store.

Perform S3 workloads

Steps

1. Deploy an S3 workload using the ObjectScale endpoints provided.

2. Verify that the S3 actions are successful.

View S3 and management certificate properties

You can view and configure the properties for S3 and management certificates.

About this task

To view the properties of an object store certificate:

Accessing data with IAM and S3 151

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Select the namespace from the namespace drop-down on the upper right corner of the ObjectScale UI.

4. Click the name of the object store containing the certificates whose properties to view or to download.

5. In the right-most pane of the object store, click Certificates tab.

The Certificates tab consists of a S3, Management, and Replication Reciever sections.

Each of the sections of the Certificates tab shows the: a. Expiration details of the CA Bundle for S3/Management and Replication services. b. Certificate signing status displayed for each service.

If Kubernetes signed certificate is being used, which is the default configuration for object stores, S3 and Management sections allow you to Download CA Bundle. The certificate can be used for establishing trusted https connections for S3 and management service.

152 Accessing data with IAM and S3

Working with ObjectScale Replication This chapter contains:

Topics:

Introduction to ObjectScale Replication Bucket Replication Policy Manage a Bucket Replication Policy using ObjectScale UI Set up ObjectScale Replication using the ObjectScale API Monitor and manage replication for an object store

Introduction to ObjectScale Replication ObjectScale Replication allows you to manage and monitor replication policies and replicate bucket data. Replication between Object Stores is in compliance with S3 protocol of AWS.

Each source bucket can be configured to replicate some or all its data to one or more destination buckets. The data that are replicated from the source bucket can be replicated based on a key prefix, a tag or both to make replication granular. An IAM role must be selected for the source bucket account to replicate the data. On source bucket, the rule can target specific destination buckets that are based on the key prefix and tag.

In ObjectScale:

Replication is bucket-to-bucket. Replication rules can be different for every bucket depending on redundancy or locality needs. Replication objects can be part of a bucket (by prefix). Able to do network throttling between different Object Stores.

Bucket Replication Policy A Bucket Replication Policy is an XML document that the user constructs and sets on a bucket.

You can create the Bucket Replication Policy XML document with the ObjectScale Portal UI or the S3 API. The XML format is defined by AWS (https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketReplication.html).

NOTE: Most of the fields are compatible with AWS, some fields (such as destination arn) are not compatible with

AWS.

When using the ObjectScale Portal UI to manage bucket replication settings, go to the Replication tab of a bucket (Object Store > Bucket > Replication ). From the Replication tab, you can manage replication policy of the bucket.

9

Working with ObjectScale Replication 153

Figure 25. Bucket replication settings

You can create a NEW REPLICATION RULE using the New Replication Rule wizard, which helps in adding a rule to the replication policy of the bucket.

In a replication rule we can have only one destination configured, we can not define multiple destination in a single replication rule. A replication rule created can have only one destination bucket configured from the UI.

Also, from the Replication tab you can edit or delete a rule, enable or disable rules, and change the priority of rules. The Receive Objects button is used to configure a bucket as a destination bucket. For more information, see Manage a Bucket Replication Policy using ObjectScale UI.

When managing policy documents using the S3 API, an example basic policy XML document might look like:

arn:aws:iam::AcctID:role/role-name rule1 1 Enabled important/ arn:

The IAM role used to perform the object replication must have permissions to replicate objects from the source bucket to the destination bucket. The filter of a replication rule can specify an optional prefix for prefix matching the object name, and an optional set of object tags to match. A single policy supports up to 1000 rules.

ObjectScale supports the standard AWS S3 APIs for getting, setting, and deleting the replication policy on a bucket.

Endpoint API Permissions Needed

PUT /?replication PutBucketReplication s3:PutReplicationConfiguratio n

154 Working with ObjectScale Replication

Endpoint API Permissions Needed

GET /?replication GetBucketReplication s3:GetReplicationConfiguratio n

DELETE /?replication DeleteBucketReplication s3:PutReplicationConfiguratio n

Amazon defines two versions of Replication Policies (V1 and V2). ObjectScale supports the V2 policy format. ObjectScale bucket replication policies support most of the V2 tags. There is no support for:

S3 RTC-related tags (Metrics and ReplicationTime), the ExistingObjectReplication tag

tags that are related to KMS-encrypted objects, and tags related to delete marker replication or replica sync.

Replication Rules

User replication configuration could include at most 1000 (default, configurable if users have more resource) rules.

Each rule can specify only one destination bucket. If there is a requirement to specify multiple destination buckets, the user has to configure multiple rules. The bucket is specified with the bucket ARN (include ObjectScale, object store and bucket name). Up to four (default,

configurable if users have more resource) destinations for single bucket replication configurations. From ObjectScale Portal UI, you can only select one destination per policy.

Each rule specifies the scope of the objects that this rule match.

The scope could be an entire bucket or object with specified prefix or specified tags. Configurations for whether to replicate KMS-encrypted objects are ignored.

Able to specify the replication behavior for each rule, like storage class REPLICA object writes on each destination, or whether REPLICA owner must be changed.

Able to specify whether a rule is disabled or enabled. Set priority for each rule. When a user creates an object and matches multiple rules for one destination, only the rule with

the highest priority takes effect. Rules for different target do not interfere with each other. Replication Behavior sets as when the object version is created.

Changes (priority, behavior, scope, and so on) on rules after creation of the object version only affects whether a further attribute update is replicated. It does not impact replication behavior for this object version (that is storage class on destination or if owner is changed).

Bucket Replication Limits

Policy property Default Maximum Description

Destinations per bucket 4 The maximum number of destinations allowed in a single bucket replication policy. These are unique destinations. While configuring replication rules you can configure four unique destinations which are configurable.

NOTE: From ObjectScale Portal UI, you can only select one destination per policy.

Rules per policy 1000 The maximum number of rules allowed in a single replication policy.

Policy size (bytes) 2 MB The maximum size of a replication policy, in bytes (2 MB allows 1000 rules per destination with about 2 KB of filters and other configuration per rule).

Working with ObjectScale Replication 155

Bucket Replication to multiple destinations

ObjectScale supports replication to multiple destinations. The limitation is that there cannot be more than one destination bucket in the same object store within a single replication policy.

Shown below is a simple example of an XML document for a bucket replication policy with multiple destinations:

arn:aws:iam::AcctID:role/jimmy rule1 1 foo arn:aws:s3:US-E:foo:bucket1 rule2 2 food arn:aws:s3:US-E:foo:bucket1 MCFREEZE rule3 3 foodo arn:aws:s3:US-W:foo:bucket2

In this example policy, there are two different destination buckets. Depending on the name of the object, some of the rules match to determine which destination buckets the object are replicated to and the parameters for that replication.

For example:

Object Name Rules Matched Destination Buckets Other Behavior

bar none no replication none

foo rule1 bucket1 none

food rule1, rule2 bucket1 The storage class of the object in bucket1 will be set to MCFREEZE because rule2 has the highest priority of all the rules that matched for destination "bucket1".

foodo rule1, rule2, rule3 bucket1 & bucket2 The storage class of the object in bucket1 will be set to MCFREEZE because rule2 has the highest priority of all the rules that matched for destination "bucket1". The storage class of the object in bucket2 will still be the bucket's default storage class.

156 Working with ObjectScale Replication

Manage a Bucket Replication Policy using ObjectScale UI ObjectScale UI can be used to manage bucket replication policies used for ObjectScale Replication.

Use the following tasks to manage a bucket replication policy using ObjectScale.

Configure a new bucket replication rule

This task describes to configure buckets rules for ObjectScale Replication.

Prerequisites

The user must have access to a bucket with versioning enabled.

Ensure that you have the Bucket Name available. You will need to type this name into the New Replication Rule wizard.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Click the name of the object store that contains the bucket to be modified.

4. In the right-most pane for the object store, click Buckets tab.

5. Click the name of the bucket to be modified.

The bucket details page appears. The Summary tab is displayed by default. NOTE: Alternatively, you can go to the bucket using the Accounts > > Buckets tab >

.

6. Click the Replication tab. The data grid displays a listing of existing replication rules. NEW REPLICATION RULE button is enabled by default. EDIT and DELETE are disabled until an existing replication rule is selected. ACTIONS drop-down menu is enabled by default, and consists of Enable Rule(s), Disable Rule(s), Edit Priority, and

Receive Objects.

7. Click the NEW REPLICATION RULE button. The NEW REPLICATION RULE window opens. The Rule tab is opened by default.

8. Fill the mandatory fields in the Rule tab.

Working with ObjectScale Replication 157

Figure 26. New Replication Rule - Rule

a. Enable versioning for the source bucket.

NOTE: This option will come only if the bucket versioning was not enabled when the bucket was created.

b. Enter Rule Name. c. Select Highest (default) or Lowest in Priority. d. Click to enable (default) or disable Rule Status. e. Click NEXT.

The Source Bucket tab opens.

9. Fill the mandatory fields in the Source Bucket tab.

158 Working with ObjectScale Replication

Figure 27. New Replication Rule - Source Bucket

a. The Source Bucket Name and Source Account fields are prepopulated. b. Select Rule Scope.

Select Entire Bucket to apply this rule to the entire bucket, or Select Prefix/Tag to choose a subset of objects with a specific key prefix and specific tags to which this rule applies.

Type the prefix value in the Enter Prefix field.

Type the tag Key and Value in the Tags fields. Each tag is a combination of Key and Value pair.

Click ADD TAG to add more tags. Click DELETE to delete a tag. A minimum of one tag should be provided.

c. Click NEXT.

The Destination Bucket tab opens.

10. Fill the mandatory fields in the Destination Bucket tab.

NOTE: You cannot configure more than one destination bucket at a time.

Working with ObjectScale Replication 159

Figure 28. New Replication Rule - Destination Bucket

a. Select Set Destination.

Select Buckets in current ObjectScale instance, or Select Buckets in remote ObjectScale instance.

b. Select the namespace from the drop-down menu next to Namespace.

The drop-down menu will display the namespaces available in the selected ObjectScale instance.

c. Select the object store from the drop-down menu next to Object Store.

The drop-down menu will display the object stores available in the selected ObjectScale instance.

d. Set the Destination Bucket Account as Choose a bucket in this account or Specify a bucket in another account. e. If you selected Specify a bucket in another account, click the Object Owner checkbox to change object ownership to

the destination Bucket owner.

160 Working with ObjectScale Replication

f. Type the name of the destination bucket in Destination Bucket field. g. Select the source account IAM role from the drop-down menu next to IAM Role from source account. h. Click NEXT.

The Review tab opens.

11. Review the fields in the Review tab and click SAVE.

Edit an existing bucket replication rule

This task describes to edit destination bucket rules that are configured for ObjectScale Replication.

Prerequisites

The user must have access to buckets with versioning enabled that are configured for ObjectScale Replication.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Click the name of the object store that contains the bucket to be modified.

4. In the right-most pane for the object store, click Buckets.

5. Click the name of the bucket to be modified.

The bucket details page appears. The Summary tab is displayed by default. NOTE: Alternatively, you can go to the bucket using the Accounts > > Buckets tab >

.

6. Click the Replication tab. The data grid displays a listing of existing replication rules. NEW REPLICATION RULE button is enabled by default. EDIT and DELETE buttons are enabled. ACTIONS drop-down menu is enabled by default, and consists of Enable Rule(s), Disable Rule(s), Edit Priority, and

Receive Objects.

7. Select the replication rule you want to edit and click the EDIT button.

The EDIT REPLICATION RULE window opens. The Rule tab is opened by default.

8. Navigate to the desired section of the edit replication rule wizard and update the desired value(s). Go to Rule, to edit the Rule Name or Rule Status. Go to Source Bucket, to edit the Rule Scope. Go to Destination Bucket, to edit the Set Destination, Namespace, Object Store, Destination Bucket Account, or IAM

Role from the source account value(s). After modifying the desired value(s) on each tab, click SAVE.

Delete a bucket replication rule

This task describes to delete replication rules that are configured for ObjectScale Replication.

Prerequisites

The user must have access to buckets with versioning enabled that are configured for ObjectScale Replication.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores:

Working with ObjectScale Replication 161

For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management. Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores.

For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Click the name of the object store that contains the bucket to be modified.

4. In the right-most pane for the object store, click Buckets.

5. NOTE: You cannot delete more than one bucket at an instance.

Click the name of the bucket to be modified.

The bucket details page appears. The Summary tab is displayed by default. NOTE: Alternatively, you can go to the bucket using the Accounts > > Buckets tab >

.

6. Click the Replication tab. The data grid displays a listing of existing replication rules. NEW REPLICATION RULE button is enabled by default. EDIT and DELETE buttons are enabled. ACTIONS drop-down menu is enabled by default, and consists of Enable Rule(s), Disable Rule(s), Edit Priority, and

Receive Objects.

7. Select the replication rule you want to remove and click the DELETE button. A confirmation window opens.

8. Click Yes to delete. The replication rule is deleted from the replication policy.

Working with bucket replication rules

This task describes to configure rules on a destination bucket that is configured for ObjectScale Replication.

Prerequisites

The user must have access to a bucket with versioning enabled that are configured for ObjectScale Replication.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Click the name of the object store that contains the bucket to be modified.

4. In the right-most pane for the object store, click Buckets.

5. Click the name of the bucket to be modified.

The bucket details page appears. The Summary tab is displayed by default. NOTE: Alternatively, you can go to the bucket using the Accounts > > Buckets tab >

.

6. Click the Replication tab. The data grid displays a listing of existing replication rules.

7. Select one or more rules for a destination bucket. The EDIT button is enabled. The DELETE button is enabled. Click the ACTIONS button to display additional actions:

The Enable Rule(s) button is enabled only if all the selected rule is in a disabled state. The Disable Rule(s) button is enabled only if all the selected rule is in an enabled state. The Edit Priority button is enabled when one or more rules are defined. The Receive Objects button is always enabled.

162 Working with ObjectScale Replication

8. Click either of the buttons. DELETE - The selected rule is deleted for the destination bucket. Enable Rule(s) - The selected rule is enabled for the destination bucket. Disable Rule(s) - The selected rule is disabled for the destination bucket.

NOTE: To EDIT the selected rule for the destination bucket, see Edit an existing bucket replication rule. To Edit

Priority of the selected rule, see Change the priority of bucket replication rules. For more information on Receive

Objects, see Configure destination bucket to receive objects.

A confirmation window opens.

9. Click: Yes to proceed with enable, disable or delete the selected rules for the destination bucket. No to cancel.

Change the priority of bucket replication rules

This task describes to change priority of rules on a destination bucket that are configured for ObjectScale Replication.

Prerequisites

The user must have access to source and destination buckets with versioning enabled that are configured for ObjectScale Replication.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Click the name of the object store that contains the bucket to be modified.

4. In the right-most pane for the object store, click Buckets.

5. Click the name of the bucket to be modified.

The bucket details page appears. The Summary tab is displayed by default. NOTE: Alternatively, you can go to the bucket using the Accounts > > Buckets tab >

.

6. Click the Replication tab. The data grid displays a listing of existing replication rules. ACTIONS drop-down menu is enabled by default, and consists of Enable Rule(s), Disable Rule(s), Edit Priority, and

Receive Objects.

7. Click Edit Priority.

The Edit Priority wizard opens.

8. Click the up or down arrow next to the replication rule to change the priority of a particular rule or rule(s).

9. After changing the priority of the desired rules, click SAVE. The storage policy is updated.

Configure destination bucket to receive objects

This task describes to configure a destination bucket receive objects.

Prerequisites

The user must have access to a destination bucket with versioning enabled that are configured for ObjectScale Replication.

Working with ObjectScale Replication 163

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Click the name of the object store that contains the bucket to be modified.

4. In the right-most pane for the object store, click Buckets.

5. Click the name of the bucket to be modified.

The bucket details page appears. The Summary tab is displayed by default. NOTE: Alternatively, you can go to the bucket using the Accounts > > Buckets tab >

.

6. Click the Replication tab. The ACTIONS drop-down menu is enabled by default, and consists of ENABLE RULE/S, DISABLE RULE/S, EDIT PRIORITY, and RECIEVE OBJECTS.

7. Click RECEIVE OBJECTS. The Receive Objects window opens.

Figure 29. Receive objects

8. Enter the Bucket Policy statement into the Bucket Policy field.

a. Click View and change the field to Text entry mode. b. Type the bucket policy statement into the text field.

For example,

{ "Version": "2012-10-17", "Statement": [ { "Action": [ "s3:GetBucketVersioning",

164 Working with ObjectScale Replication

"s3:ObjectOwnerOverrideToBucketOwner", "s3:ReplicateObject", "s3:ListBucketVersions", "s3:ReplicateTags", "s3:PutObjectRetention", "s3:PutObjectLegalHold", "s3:BypassGovernanceRetention" ], "Resource": [ "arn:aws:s3:${TARGET_SCALE_ID}:${TARGET_STORE_ID}:${TARGET_BUCKET_NAME}", "arn:aws:s3:${TARGET_SCALE_ID}:${TARGET_STORE_ID}:${TARGET_BUCKET_NAME}/*" ], "Effect": "Allow", "Principal": { "AWS": [ "${CRR_ROLE_ARN}", "urn:osc:iam::${SOURCE_ACCOUNT_ID}:root" ] } } ] }

c. Click Text and change the field to View mode. d. Use the toggle to enable/disable Bucket Versioing.

Bucket Version should be enabled before setting Replication configurations.

9. Click SAVE. The destination bucket starts to receive replicated objects from the source bucket.

Set up ObjectScale Replication using the ObjectScale API Before you can set up ObjectScale Replication between two ObjectScale instances within a federation using the ObjectScale API, you must first have completed the following prerequisites:

1. Installed the primary ObjectScale instance and created an object store. 2. Installed the secondary ObjectScale instance and created an object store. 3. Created an ObjectScale federation consisting of these two ObjectScale instances.

After you have completed these prerequisites, you can now set up ObjectScale Replication. To set up ObjectScale Replication using the ObjectScale API, do the following:

1. Create and configure an account and an IAM role 2. Setup the ObjectScale to ObjectScale Replication

Create and configure an account and an IAM role

About this task

NOTE: The $OSR_ROLE_ARN in replication configuration can take any valid service role ARN. Create an IAM role and give

permission to enable replication.

Steps

1. Create a global account.

Working with ObjectScale Replication 165

a. Set the environment variables and display the Account ID:

IAMSVC_ENDPOINT=$(kubectl get svc | awk '/-iam\s/{print $3}' )

FEDSVC_ENDPOINT=$(kubectl get svc | awk '/fedsvc\s/{print $3}' )

TOKEN=$(curl -ik -u root:ChangeMe http://$FEDSVC_ENDPOINT:9500/mgmt/login | awk '/ X-SDS-AUTH-TOKEN/{print $2; exit}')

TOKEN=${TOKEN//[$'\r\n']}

b. Create the new account and display the Account ID.

ACCOUNT_ID=$(curl -X POST http://${IAMSVC_ENDPOINT}:9400/iam?'Action=CreateAccount' -H "X-SDS-AUTH-TOKEN:$TOKEN" -v | xmllint --format - | grep 'AccountId' | sed 's/ \(.*\)<\/AccountId>/\1/g' |sed -e 's/^[ \t]*//')

# output: ACCOUNT_ID="a7bf6bfe35ac4277a1a8857da98b3226"

2. Create an IAM role under the global account.

a. Set the environment variables for the new role name:

OSR_ROLE_NAME="osrRole2"

b. Create an IAM role under the global account.

OSR_ROLE_ARN=$(curl http://${IAMSVC_ENDPOINT}:9400/ iam?'Action=CreateRole&RoleName='$OSR_ROLE_NAME'&MaxSessionDuration=43200&AssumeRole PolicyDocument=%7B%22Version%22%3A%222012-10-17%22%2C%22Statement%22%3A%5B%7B%22Effe ct%22%3A%22Allow%22%2C%22Principal%22%3A%7B%22Service%22%3A%22crr.objscale.dell.com% 22%7D%2C%22Action%22%3A%22sts%3AAssumeRole%22%7D%5D%7D' -H "x-emc- namespace:$ACCOUNT_ID" -H "X-SDS-AUTH-TOKEN:$TOKEN" -v | xmllint --format - | grep 'Arn' | sed 's/ \(.*\)<\/Arn>/\1/g' |sed -e 's/^[ \t]*//')

# output: OSR_ROLE_ARN="urn:osc:iam::a7bf6bfe35ac4277a1a8857da98b3226:role/osrRole2"

The AssumeRolePolicyDocument is URL encoded JSON. { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "crr.objscale.dell.com" }, "Action": "sts:AssumeRole" } ] } This policy allow ObjectScale Replication services to assume this role.

3. Create a policy to attach to a role.

a. Set the environment variables for the new policy name:

OSR_POLICY_NAME="osrPolicy"

b. Create a policy to attach to a role.

OSR_POLICY_ARN=$(curl http://${IAMSVC_ENDPOINT}:9400/ iam?'Action=CreatePolicy&PolicyName='$OSR_POLICY_NAME'&PolicyDocument=%7B%22Version% 22%3A%222012-10-17%22%2C%22Statement%22%3A%5B%7B%22Effect%22%3A%22Allow%22%2C%22Acti on%22%3A%22s3%3A*%22%2C%22Resource%22%3A%5B%22*%22%5D%7D%5D%7D' -H "x-emc-

166 Working with ObjectScale Replication

namespace:$ACCOUNT_ID" -H "X-SDS-AUTH-TOKEN:$TOKEN" -v | xmllint --format - | grep 'Arn' | sed 's/ \(.*\)<\/Arn>/\1/g' |sed -e 's/^[ \t]*//')

The policy allows permissions on all S3 actions:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "s3:*", "Resource": [ "*" ] } ] }

4. Attach the policy to a role.

NOTE: The role will have the permission only after you have attached the policy to it.

curl http://${IAMSVC_ENDPOINT}:9400/ iam?'Action=AttachRolePolicy&RoleName='$OSR_ROLE_NAME'&PolicyArn='$OSR_POLICY_ARN -H "x-emc-namespace:$ACCOUNT_ID" -H "X-SDS-AUTH-TOKEN:$TOKEN" -v | xmllint --format

The role can now be used as replication role in replication configuration.

Setup the ObjectScale to ObjectScale Replication

Steps

1. Create source and destination bucket on the ObjectScale instances. You must enable versioning on both of the buckets.

2. Attach the bucket policy to the source and destination buckets to ensure that the proper privileges are provided to the account and IAM service role.

3. Use the ObjectScale fedsvc API to GET the provisioned object store:

a. Set the environment variables:

IAMSVC_ENDPOINT=$(kubectl get svc | awk '/-iam\s/{print $3}' )

FEDSVC_ENDPOINT=$(kubectl get svc | awk '/fedsvc\s/{print $3}' )

TOKEN=$(curl -ik -u root:ChangeMe http://$FEDSVC_ENDPOINT:9500/mgmt/login | awk '/ X-SDS-AUTH-TOKEN/{print $2; exit}')

TOKEN=${TOKEN//[$'\r\n']}

b. Use curl to issue the GET objectstores call:

curl -k -X GET http://$FEDSVC_ENDPOINT:9500/fedsvc/objectstores -H "Content-Type: application/xml" -H "X-SDS-AUTH-TOKEN:$TOKEN" | xmllint --format -

OSTI5C2F43D1FF525835 OSCI6081476846ED9A56 default ecs-cluster ecs.dellemc.com/v1beta1 0.71.2 Available

Working with ObjectScale Replication 167

false 2021-04-22T10:09:06Z 15d61162-d218-4069-b09f-aaa86213098e

4. Use the ObjectScale API to PUT the replication configuration. In this configuration you will need to specify the ObjectScale and object store ID in the target bucket ARN

For example:

SRC_BUCKET_NAME="source1" # on object scale 1 DEST_BUCKET_NAME="destination1" # on object scale 2 SCALE_ID="OSCIFFFFFFFFEBC3958D" # object scale 2 object scale id STORE_ID="OSTIE5BDD14DBC63185E" # object scale 2 object store id OSR_ROLE_ARN="urn:osc:iam::a7bf6bfe35ac4277a1a8857da98b3226:role/crrRole2" rm -f $HOME/.osr-rep-config cat >> $HOME/.osr-rep-config << EOF $OSR_ROLE_ARN Enabled 1 arn:aws:s3:$SCALE_ID:$STORE_ID:$DEST_BUCKET_NAME EOF $HOME/s3curl/s3curl.pl --id=ecsflex --calculateContentMd5 --put=$HOME/.osr-rep-config -- -v http://$(kubectl get svc | awk '/-s3/{print $3}')/$SRC_BUCKET_NAME?replication $HOME/s3curl/s3curl.pl --id=ecsflex -- -v http://$(kubectl get svc | awk '/-s3/{print $3}')/$SRC_BUCKET_NAME?replication

Monitor and manage replication for an object store This task describes how monitor and manage ObjectScale Replication for an object store.

Prerequisites

The object store must not contain any buckets with replication rules configured. The user must have access to an object store that contains source buckets with replication configured.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Click the name of the object store.

4. Click Replication. Three cards are displaying replication metrics aggregated for all destination object stores.

Data In (All) Data yet to be replicated (24 hours) Failed Objects (24 hours)

A table is displayed that consists of the object stores that are configured as replication destinations. These object stores can be managed in the following ways: THROTTLE : This operation would limit the replication rate from source object store to selected object stores.

168 Working with ObjectScale Replication

UNTHROTTLE : This operation would remove the limit on replication rate from source object store to selected object stores.

PAUSE : This operation would pause the replication from source object store to selected object stores for certain duration.

SUSPEND : This operation would suspend the replication from source object store to selected object stores. RESUME : This operation would recover from both PAUSE and SUSPEND.

5. Select an object store.

If selected object store is not be in, paused or suspended state:

The THROTTLE, UNTHROTTLE, PAUSE, and SUSPEND buttons are enabled.

If selected object store is in, paused or suspended state:

The RESUME buttons are enabled.

6. Click either of the buttons. THROTTLE, or UNTHROTTLE, or PAUSE, or SUSPEND, or RESUME.

Table 35. Working with ObjectScale Replication at Object Store

Action Result

Click THROTTLE > SAVE. The user is allowed to throttle the data that is being replicated to the selected object store by supplying a single numeric value that represents MB/s.

CANCEL and SAVE buttons are enabled.

Click UNTHROTTLE > SAVE. The user is allowed to unthrottle the data that is being replicated to the selected object store by supplying a single numeric value that represents MB/s.

CANCEL and SAVE buttons are enabled.

a. Click PAUSE, and fill the required fields. b. Click APPLY.

a. An overhead incurred estimate is displayed to perform the pause operation.

The APPLY button is enabled after you click the checkbox acknowledging the understanding of overhead incurred.

The CANCEL button is enabled. b. The replication data flowing to the object store is

paused. Objects that are created in the source bucket during

the pause duration are replicated upon resume. The status column in the object store row changes to

PAUSED.

Click SUSPEND > YES. The replication data that is flowing to the object store is suspended.

Objects that are created in the source bucket during the suspend duration are not replicated upon resume.

The status column in the object store row changes to SUSPEND.

Click RESUME > YES. The object stores that are either paused or suspended resumes being replicated to the destination buckets.

The status column in the object store changes to Running.

NOTE: For details on the ObjectScale Replication Control APIs available in this release, see the ObjectScale REST API

zip file posted at https://www.dell.com/support/home/product-support/product/objectscale/drivers.

Working with ObjectScale Replication 169

Monitoring Events: Audits and Alerts Monitoring events provides information about the monitoring messages in the UI.

Topics:

About ObjectScale instance event and issue monitoring Monitoring Events, Audits, and Alerts

About ObjectScale instance event and issue monitoring Throughout ObjectScale there are processes that are constantly monitoring and collecting information on the ObjectScale instance and object stores. When the status of a component or operation changes, the change is captured and noted in the following places in ObjectScale:

ObjectScale > Health

NOTE: In the vSphere Client this page is located at Workload Cluster > Monitor > ObjectScale > Health.

Object Stores > > Health NOTE: In the vSphere Client this page is located at Workload Cluster > Configure > ObjectScale > Object Stores >

> Health.

ObjectScale issues and event data is available to be filtered for the last 24 hours, last seven days, last one month, or all.

NOTE: Issues with the description pods are not scheduled due to insufficient resources may appear in

the Health tab even though all pods are up and running. The kubernetes FailedScheduling event gets generated during

successful object store creations as well sometimes as the creation is in progress. These issues can be ignored and will get

immediately cleared after the pod is scheduled.

About alert severities

Event and issue alert message severity labels have the following meanings:

Critical: Messages about conditions that require immediate attention. Error: Messages about error conditions that report either a physical failure or a software failure. Warning: Messages about less than optimal conditions. Normal: Routine status messages. Audit: Audit messages for events only.

View ObjectScale health issues and events

View health issues and events for the ObjectScale instance at:

For vSphere Client UI: Workload Cluster > Monitor > ObjectScale > Health For Portal UI: ObjectScale > Health

10

170 Monitoring Events: Audits and Alerts

Figure 30. ObjectScale heath issues and events

The Health page contains the full list of current health issues and health events for the ObjectScale instance.

To review health issues, select Issues to display the full list of current issues. You can filter the issues listed using the dropdown above the table. You can filter issues from the last 24 hours, last 7 days, last month, or all.

You can also filter each individual column using the filter icon.

Select an issue listed in the table. If the issue type is Manual, use Acknowledged or Unacknowledged to modify an issue once reviewed.

To review health events, select Events to display the full list of current events. You can filter the events listed using the dropdown above the table. You can filter events from the last 24 hours, last 7 days, last month, or all.

To review topology health, select Topology to display the full list of the Kubernetes storage.

View the health of an object store

The health issues and events page for individual object stores can be found at:

For vSphere Client UI: Workload Cluster > Configure > ObjectScale > Object Store > > Health For Portal UI: Object Store > > Health

Click ..., if the Health tab is not visible.

Monitoring Events: Audits and Alerts 171

Figure 31. Object store heath issues, events, and health check

The object store's Health tab displays the full list of current health issues and health events for the selected object store.

The health Issues tab displays the object store's issues. Health issues are placed into one of two categories, Auto or Manual. You can use the ACKNOWLEDGE or UNACKNOWLEDGE buttons to manage manual health issues.

Auto issues are the issues generated within the product if a particular component does not behave as expected and will get cleared automatically once the problem is resolved.

Manual issues are the issues that will not get be cleared until acknowledged by a user.

The health Events tab shows the full list of current events.

The Health Check tab allows you to perform a variety of health checks on the object store.

Select healthcheck and click Check Health to perform a check on the health of the object store. Select pre-update and click Check Health to perform a pre-update health check of the object store before updating the

object store. Select click Check Health to perform a health check of the object store following an upgrade.

Monitoring Events, Audits, and Alerts

CSI-01

Name DriveHealthFailure Severity level Error Description Drive health is BAD, previous state: . Drive Details: SN= ,

Model= , Type= , Size= , Node= , Firmware=

Resource ID Drive

KAHM issue DiskHealthIssue Clearance Auto

CSI-01

Name DriveHealthSuspect Severity level Warning Description Drive health is SUSPECT, previous state: . Drive Details: SN= ,

Model= , Type= , Size= , Node= , Firmware=

172 Monitoring Events: Audits and Alerts

Resource ID Drive

KAHM issue DiskHealthIssue Clearance Auto

CSI-01

Name DriveReadyForRemoval Severity level Warning Description Drive is ready for documented removal procedure. Drive Details: SN= ,

Model= , Type= , Size= , Node= , Firmware=

Resource ID Drive

KAHM issue DiskHealthIssue Clearance Auto

CSI-01

Name DriveReadyForPhysicalRemoval Severity level Warning Description Drive successfully removed from CSI, and ready for physical removal,

Drive Details: SN= , Model= , Type= , Size= , Node= , Firmware=

Resource ID Drive

KAHM issue DiskHealthIssue Clearance Auto

CSI-01

Name DriveSuccessfullyRemoved Severity level Normal Description Drive successfully removed. Drive status is: OFFLINE, previous status:

ONLINE. Drive Details: SN= , Model= , Type= , Size= , Node= , Firmware=

Resource ID Drive

KAHM issue DiskHealthIssue Clearance Auto

CSI-01

Name DriveRemovalFailed Severity level Error Description Failed to locale LED, Drive Details: SN= , Model= ,

Type= , Size= , Node= , Firmware=

Monitoring Events: Audits and Alerts 173

Failed to remove volume with error: . Drive Details: SN= , Model= , Type= , Size= , Node= , Firmware=

Failed to release volume(s), Drive Details: SN= , Model= , Type= , Size= , Node= , Firmware=

Resource ID Drive

KAHM issue DiskHealthIssue Clearance Auto

CSI-03

Name DriveStatusOffline Severity level Error Description Drive status is: OFFLINE, previous status: ONLINE. Drive Details: SN= ,

Model= , Type= , Size= , Node= , Firmware=

Resource ID Drive

KAHM issue DiskMissing Clearance Auto

CSI-03

Name DriveStatusOnline Severity level Normal Description Drive status is: ONLINE, previous status: OFFLINE. Drive Details: SN= ,

Model= , Type= , Size= , Node= , Firmware=

Resource ID Drive

KAHM issue DiskMissing Clearance Auto

CSI-04

Name DriveHealthGood Severity level Normal Description Drive health is GOOD, previous state: . Drive Details: SN= ,

Model= , Type= , Size= , Node= , Firmware=

Resource ID Drive

KAHM issue Informative event.

Clearance -

174 Monitoring Events: Audits and Alerts

CSI-05

Name FakeAttachInvolved Severity level Error Description Fake-attach involved for volume with ID Resource ID Volume

KAHM issue VolumeFakeAttach Clearance Auto

CSI-05

Name FakeAttachCleared Severity level Normal Description Fake-attach cleared for volume with ID Resource ID Volume

KAHM issue VolumeFakeAttach Clearance Auto

DECKS-HC-1000

Name Pre-Update Description Preupdate health check for application.

Issue Category Auto

Notifiers objectscale-snmp-notifier Remedies --

DECKS-LIC-1002

Name ExpiringLicense Description License is expiring or expired.

Issue Category Auto

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies --

DECKS-LIC-1005

Name ExpiringLicense Description License is expiring or expired.

Issue Category Auto

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Check the end date of the ObjectScale license.

Monitoring Events: Audits and Alerts 175

Go to the Dell EMC Software Licensing Center (SLC) to renew or extend the ObjectScale license. Contact your Dell EMC sales representative to renew or extend the ObjectScale license. For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

DECKS-LIC-1006

Name ExpiringLicense Description License is expiring or expired.

Issue Category Auto

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Check the end date of the ObjectScale license. Go to the Dell EMC Software Licensing Center (SLC) to renew or extend the ObjectScale license. Contact your Dell EMC sales representative to renew or extend the ObjectScale license. For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

DECKS-LIC-1008

Name InvalidLicense Description License is invalid.

Issue Category Auto

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Verify that the ObjectScale license is obtained from the Dell EMC Software Licensing Center. Verify that the ObjectScale license is not modified before applying it to the cluster. Verify that the PRODUCTSHORTNAME is defined in the ObjectScale license.

For more information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search for the knowledge base article.

DECKS-LIC-1011

Name - Description License features are no longer tracked.

Issue Category -

Notifiers - Remedies Verify that the ObjectScale license is correct and the feature was intended to be removed.

For more information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search for the knowledge base article.

DECKS-SA-1023

Name SupportAssistConfiguration Description SupportAssist configuration issues.

176 Monitoring Events: Audits and Alerts

Issue Category Auto

Notifiers objectscale-snmp-notifier Remedies Verify that the supportassist-objectscale-0 pod is running.

Verify that SupportAssist is enabled. Verify connectivity of configured gateways. Verify that a valid AccessKey and PIN are used. For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

DECKS-SA-1024

Name ESECallBackTransactions Description SupportAssist ESE callback transaction issues.

Issue Category Auto

Notifiers objectscale-snmp-notifier Remedies Verify the supportassist-objectscale-0 pod is Running

Verify connectivity of configured gateways. Check network connectivity of the k8s cluster Check the log of the supportassist-objectscale-0 pod For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

KAHM-HC-1000

Name Pre-Update Description Pre-Update health check for application.

Issue Category Auto

Notifiers objectscale-snmp-notifier Remedies Check the health status in the -app-configmap to find which checks failed.

For more information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search for the knowledge base article.

OBJSC-LIC-0004

Name ObjectScale Licensing Description ObjectScale cumulative object store usage.

Issue Category Auto

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies If near or above capacity, remove stale object stores. If near or above capacity, contact Dell Technologies for an updated license. For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

Monitoring Events: Audits and Alerts 177

OBJSC-MGR-3000

Name Update Description Update for application.

Issue Category Auto

Notifiers objectscale-snmp-notifier Remedies Check the health status in the -app-configmap to find which checks failed.

For more information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search for the knowledge base article.

OBJSC-MGR-HC-1000

Name Pre-Update Description Pre-Update health check for application.

Issue Category Auto

Notifiers objectscale-snmp-notifier Remedies Check the health status in the -app-configmap to find which checks failed.

For more information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search for the knowledge base article.

OBJSC-MON-1111

Name Objectscale Capacity Description Percent of used ObjectScale capacity crosses threshold.

Issue Category Auto, 60

Notifiers objectscale-snmp-notifier Remedies Verify ObjectScale capacity usage, as required take proactive actions to prevent ObjectScale

maximum capacity usage. For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJSC-MON-1112

Name Objectscale Capacity Description Percent of used ObjectScale capacity crosses threshold.

Issue Category Auto, 60

Notifiers objectscale-snmp-notifier Remedies Verify ObjectScale capacity usage, as required take proactive actions to prevent ObjectScale

maximum capacity usage. For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJSC-MON-1113

Name Objectscale Capacity

178 Monitoring Events: Audits and Alerts

Description Percent of used ObjectScale capacity crosses threshold.

Issue Category Auto, 60

Notifiers objectscale-snmp-notifier Remedies Verify ObjectScale capacity usage, as required take proactive actions to prevent ObjectScale

maximum capacity usage. For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJSC-MON-3002

Name Directory Table failure Description Directory Table failure detected

Issue Category Auto, 10

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Contact DellEMC Support for additional information For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJSC-MON-3003

Name Directory Table failure Description Directory Table failure detected

Issue Category Auto, 10

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Check if user application is fully available Contact DellEMC Support for additional information For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJSC-MON-4019

Name Objectscale Monitoring Health Description No data is moved to the ObjectScale monitoring framework for the last 30 minutes.

Issue Category Auto, 60

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Check monitoring components health (telegraf, influxdb, fluxd) For more information about this event, go to https://www.dell.com/support/kbdoc/en-us/

000195833 and use the SymptomID to search for the knowledge base article.

OBJSC-MON-4020

Name MonitoringFluxd

Monitoring Events: Audits and Alerts 179

Description Fluxd has not responded for the last 30 minutes.

Issue Category Auto, 60

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Check Fluxd service status.

For more information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search for the knowledge base article.

OBJSC-MON-4021

Name InfluxDB PVC Description InfluxDB PVC has a bad state for the last 30 minutes.

Issue Category Auto, 60

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Check InfluxDB PVC status and fix it.

For more information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search for the knowledge base article.

OBJSC-MON-4022

Name Rsyslog PVC Description Rsyslog PVC has a bad state for the last 30 minutes.

Issue Category Auto, 60

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Check Rsyslog PVC status and fix it.

For more information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search for the knowledge base article.

OBJSC-MON-4025

Name InfluxDB low disk space Description Percent of used InfluxDB capacity crosses threshold.

Issue Category Auto, 60

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Based on capacity usage, InfluxDB may be in read-only mode. Verify InfluxDB capacity usage and take required actions to free up or increase capacity.

For more information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search for the knowledge base article.

180 Monitoring Events: Audits and Alerts

OBJSC-MON-4028

Name Rsyslog low disk space Description Percent of used Rsyslog capacity crosses threshold.

Issue Category Auto, 60

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Verify Rsyslog capacity usage and take required actions to free up capacity.

For more information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search for the knowledge base article.

OBJSC-SP-0000

Name SPOperatorRecovery Description Recovery service procedure handling by SP Operator

Issue Category Manual

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Check recovery service procedure CR in the K8s and SP operator logs for details For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJSC-SP-0001

Name SPOperatorDR Description Disk Replacement service procedure handling by SP Operator

Issue Category Auto

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Check disk replacement service procedure CR in the K8s and SP operator logs for details For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJSC-SP-0002

Name SPOperatorPMM Description Permanent Maintenance Mode service procedure handling by SP Operator

Issue Category Auto

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Check PMM service procedure CR in the K8s and SP operator logs for details For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

Monitoring Events: Audits and Alerts 181

OBJSC-SP-0003

Name SPOperatorTMM Description Temporary Maintenance Mode service procedure handling by SP Operator

Issue Category Auto

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Check TMM service procedure CR in the K8s and SP operator logs for details For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJSOP-1000

Name OperatorDR Description Disk Replacement service procedure handling by Operator

Issue Category Auto

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Check Object Store Status, Operator logs and platform logs (if applicable) for details For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJSOP-1001

Name OperatorPMM Description Permanent Maintenance Mode service procedure handling by Operator

Issue Category Auto

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Check Object Store Status, Operator logs and platform logs (if applicable) for details For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJSOP-1002

Name OperatorTMM Description Temporary Maintenance Mode service procedure handling by Operator

Issue Category Auto

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Check Object Store Status, Operator logs and platform logs (if applicable) for details For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

182 Monitoring Events: Audits and Alerts

OBJSOP-1003

Name OperatorUpgrade Description Upgrade service procedure handling by Operator

Issue Category Auto

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Check Object Store Status, Operator logs and platform logs (if applicable) for details For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJSOP-1004

Name OperatorHorizontalExpand Description Horizontal Expand SS service procedure handling by Operator

Issue Category Auto

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Check Object Store Status, Operator logs and platform logs (if applicable) for details For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJSOP-1005

Name OperatorVerticalExpand Description Vertical Expand SS service procedure handling by Operator

Issue Category Auto

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Check Object Store Status, Operator logs and platform logs (if applicable) for details For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJSOP-1006

Name OperatorObjectStoreCreation Description Object Store Creation

Issue Category Auto

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Please check Object Store Status, Object Scale Operator logs and platform logs (if applicable) for details

Please check Object Store pods that remain in Pending state Please try to eliminate errors that block pods scheduling

Monitoring Events: Audits and Alerts 183

For more information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search for the knowledge base article.

OBJSOP-2001

Name CSRRequestAndApprovals Description ObjectScale CSR issue and approval notifications

Issue Category Auto

Notifiers objectscale-snmp-notifier

Remedies Approve the certificate signing request if the CSR is pending Run the command "kubectl get csr" and it will show which CSR(s) are pending For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJSOP-2002

Name TLSCertExpire Description ObjectScale TLS certificate about to expire notification or expired notifications

Issue Category Auto

Notifiers objectscale-snmp-notifier

Remedies Check the expiration dates of the certificates Renew the certificate(s) before they expire For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-1006

Name BUCKET_HARD_QUOTA_EXCEEDED Description Hard quota on total object count or size that is exceeded for one bucket.

Issue Category Manual

Notifiers objectscale-snmp-notifier Remedies Increase hard count or size quota for this bucket or delete objects in this bucket.

For more information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search for the knowledge base article.

OBJST-1008

Name BUCKET_SOFT_QUOTA_EXCEEDED Description Soft quota on total object count or size that is exceeded for one bucket.

Issue Category Manual

Notifiers objectscale-snmp-notifier Remedies Increase soft count or size quota for this bucket or delete objects in this bucket.

For more information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search for the knowledge base article.

184 Monitoring Events: Audits and Alerts

OBJST-12001

Name REPLICATION_DESTINATION_PAUSED_WITH_BACKLOG Description ObjectScale Replication is paused and there are pending objects waiting for replication.

Issue Category Auto, 120

Notifiers objectscale-snmp-notifier Remedies Confirm if replication paused setting for reported destination is still required.

Resume replication if pause is no longer required. For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-12003

Name REPLICATION_DESTINATION_REMOVED_FROM_SYSTEM Description A destination object store for ObjectScale Replication has been removed from federation.

Issue Category Manual

Notifiers objectscale-snmp-notifier Remedies Remove ObjectScale Replication configuration targeting reported destination object store.

Suspend ObjectScale Replication to reported destination object store. For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-12004

Name REPLICATION_DESTINATION_BUCKET_QUOTA_EXCEEDED Description Destination bucket exceeds user-configured quota.

Issue Category Auto, 30

Notifiers objectscale-snmp-notifier Remedies Modify the destination bucket quota size.

Clean up some of the data in destination bucket to free up space for replication to continue. For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-12005

Name REPLICATION_DESTINATION_OBJECT_STORE_OUT_OF_CAPACITY Description Destination object store is out of capacity.

Issue Category Auto, 30

Notifiers objectscale-snmp-notifier Remedies Add more space to the destination object store.

Clean up some of the data in destination object store to free up space for replication to continue. For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

Monitoring Events: Audits and Alerts 185

OBJST-12006

Name REPLICATION_CERTIFICATE_ERROR Description Connection to remote replication endpoint cannot be established due to tls problem

Issue Category Auto, 10

Notifiers objectscale-snmp-notifier Remedies Verify if internal certificate and CA used for geo connection has expired.

Contact Dell EMC technical support for assistance For additional information on this event, go to https://dell.com/support/objectscale and use the

SymptomID to search support for the knowledge base article.

OBJST-12007

Name REPLICATION_EVENT_UNABLE_TO_HANDLE Description Unable to Handle ObjectScale Replication Event

Issue Category Manual

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies Contact Dell EMC technical support for assistance For additional information on this event, go to https://dell.com/support/objectscale and use the

SymptomID to search support for the knowledge base article.

OBJST-13000

Name STORAGE_TIER_UNAVAILABLE Description Storage tier is not available or degraded.

Issue Category Auto

Notifiers objectscale-snmp-notifier Remedies Check if all the nodes are in maintenance mode.

Check if one or more of the nodes have been powered off. Check if one or more pods are not in the Running state. Verify that the disks attached to each of the nodes are all mounted. For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-13001

Name FAULT_DOMAIN_UNAVAILABLE Description Fault domain is not available or degraded.

Issue Category Auto, 3

Notifiers objectscale-snmp-notifier Remedies Check if all the nodes are in maintenance mode.

Check if one or more of the nodes have been powered off. Check if one or more pods are not in the Running state. Verify that the disks attached to each of the nodes are all mounted.

186 Monitoring Events: Audits and Alerts

For more information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search for the knowledge base article.

OBJST-13002

Name PRAVEGA_THRESHOLD_EXCEEDED Description Pravega capacity threshold exceeded.

Issue Category Auto

Notifiers objectscale-snmp-notifier Remedies Verify on the object store dashboard under Data Management unreclaimable and reclaimable metadata

or data values are big due to possible delays in space reclamation. Verify if used capacity has exceeded defined thresholds. Verify if any hardware (nodes or disks) is down that could be causing less usable storage. For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-13003

Name CAPACITY_USAGE_FOR_OBJECT_STORE Description Capacity usage for object store.

Issue Category Auto, 60

Notifiers objectscale-snmp-notifier Remedies Verify on the object store dashboard under Data Management unreclaimable and reclaimable metadata

or data values are big due to possible delays in space reclamation. Verify if used capacity has exceeded defined thresholds. Verify if any hardware (nodes, disks) is down that could be causing less usable storage. For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-13004

Name CAPACITY_USAGE_FOR_OBJECT_STORE_ERROR Description Capacity usage for object store error.

Issue Category Auto

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies Verify on the object store dashboard under Data Management unreclaimable and reclaimable metadata

or data values are big due to possible delays in space reclamation. Verify if used capacity has exceeded defined thresholds. Verify if any hardware (nodes or disks) is down that could be causing less usable storage. For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-13005

Name CAPACITY_USAGE_FOR_OBJECT_STORE Description Capacity usage for object store.

Issue Category Auto, 60

Monitoring Events: Audits and Alerts 187

Notifiers objectscale-snmp-notifier Remedies Verify on the object store dashboard under Data Management unreclaimable and reclaimable metadata

or data values are big due to possible delays in space reclamation. Verify if used capacity has exceeded defined thresholds. Verify if any hardware (nodes or disks) is down that could be causing less usable storage. For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-13006

Name CAPACITY_USAGE_FOR_OBJECT_STORE_ERROR Description Capacity usage for object store error.

Issue Category Auto

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies -

OBJST-13007

Name STORAGE_TIER_UNAVAILABLE Description Storage tier is degraded or possibly unavailable

Issue Category Auto

Notifiers objectscale-snmp-notifier Remedies -

OBJST-13008

Name FAULT_DOMAIN_UNAVAILABLE Description Fault domain is degraded or possibly unavailable

Issue Category Auto

Notifiers objectscale-snmp-notifier Remedies -

OBJST-13009

Name PRAVEGA_CAPACITY_THRESHOLD_EXCEEDED Description Pravega capacity threshold exceeded.

Issue Category Auto

Notifiers objectscale-snmp-notifier Remedies For additional information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search support for the knowledge base article.

OBJST-1320

Name BTREE_CHUNK_SR_THRESHOLD_EXCEEDED

188 Monitoring Events: Audits and Alerts

Description System metadata space reclamation throughput is too slow to catch up with garbage detection.

Issue Category Auto, 1440

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-1321

Name BTREE_CHUNK_SR_THRESHOLD_EXCEEDED Description System metadata space reclamation throughput is too slow to catch up with garbage detection.

Issue Category Auto, 1440

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies Contact Dell EMC Support for additional information.

For additional information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search support for the knowledge base article.

OBJST-1324

Name BTREE_DISK_SR_THRESHOLD_EXCEEDED Description Capacity free-up throughput is too slow to catch up with system metadata space reclamation.

Issue Category Auto, 1440

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-1325

Name BTREE_DISK_SR_THRESHOLD_EXCEEDED Description Capacity free-up throughput is too slow to catch up with system metadata space reclamation.

Issue Category Auto, 1440

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies Contact Dell EMC Support for additional information.

For additional information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search support for the knowledge base article.

OBJST-1328

Name BTREE_PARTIAL_SR_THRESHOLD_EXCEEDED Description Partial space reclamation for system metadata is too slow.

Issue Category Auto, 1440

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

Monitoring Events: Audits and Alerts 189

OBJST-1329

Name BTREE_PARTIAL_SR_THRESHOLD_EXCEEDED Description Partial space reclamation for system metadata is too slow.

Issue Category Auto, 1440

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies Contact Dell EMC Support for additional information.

For additional information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search support for the knowledge base article.

OBJST-1332

Name REPO_CHUNK_SR_THRESHOLD_EXCEEDED Description User space reclamation throughput is too slow to catch up with garbage detection.

Issue Category Auto, 1440

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-1333

Name REPO_CHUNK_SR_THRESHOLD_EXCEEDED Description User space reclamation throughput is too slow to catch up with garbage detection.

Issue Category Auto, 1440

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies Contact Dell EMC Support for additional information.

For additional information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search support for the knowledge base article.

OBJST-1336

Name REPO_DISK_SR_THRESHOLD_EXCEEDED Description Capacity free-up throughput is too slow to catch up with user space reclamation.

Issue Category Auto, 1440

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-1337

Name REPO_DISK_SR_THRESHOLD_EXCEEDED Description Capacity free-up throughput is too slow to catch up with user space reclamation.

Issue Category Auto, 1440

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier

190 Monitoring Events: Audits and Alerts

Remedies Contact Dell EMC Support for additional information. For additional information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search support for the knowledge base article.

OBJST-1340

Name REPO_PARTIAL_SR_THRESHOLD_EXCEEDED Description Partial space reclamation for user garbage is too slow.

Issue Category Auto, 1440

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-1341

Name REPO_PARTIAL_SR_THRESHOLD_EXCEEDED Description Partial space reclamation for user garbage is too slow.

Issue Category Auto, 1440

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies Contact Dell EMC technical support for assistance.

For additional information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search support for the knowledge base article.

OBJST-1344

Name SR_STATUS_THRESHOLD_EXCEEDED Description Space reclamation for user data or system metadata is disabled.

Issue Category Auto, 1440

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-1345

Name SR_STATUS_THRESHOLD_EXCEEDED Description Space reclamation for user data or system metadata is disabled.

Issue Category Auto, 1440

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies Contact Dell EMC technical support for assistance.

For additional information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search support for the knowledge base article.

OBJST-1352

Name MEMORY_TABLE_FREE_SPACE_PERCENT

Monitoring Events: Audits and Alerts 191

Description Directory Table memory tension detected.

Issue Category Auto, 60

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-1354

Name MEMORY_TABLE_FREE_SPACE_PERCENT Description Directory Table memory tension detected.

Issue Category Auto, 60

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies Check if user application is fully available and throttle load if application is reporting errors for storage

system. Contact Dell EMC Support for additional information. For additional information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search support for the knowledge base article.

OBJST-1364

Name LISTING_CONVERSION_THRESHOLD_EXCEEDED Description Listing Conversion speed is slow.

Issue Category Auto, 60

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-1365

Name LISTING_CONVERSION_THRESHOLD_EXCEEDED Description Listing Conversion speed is slow.

Issue Category Auto, 60

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies For additional information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search support for the knowledge base article.

OBJST-1366

Name LISTING_CONVERSION_THRESHOLD_EXCEEDED Description Listing Conversion speed is slow.

Issue Category Auto, 60

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies For additional information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search support for the knowledge base article.

192 Monitoring Events: Audits and Alerts

OBJST-1390

Name SSD_READ_CACHE_CAPACITY_FAILURE Description SSD read cache auto cleanup fails when capacity full and fallback to memory cache.

Issue Category Auto, 60

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-1392

Name SSD_READ_CACHE_CAPACITY_FAILURE Description SSD read cache auto cleanup fails when capacity full and fallback to memory cache.

Issue Category Auto, 60

Notifiers objectscale-supportassist-ese objectscale-snmp-notifier

Remedies For additional information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search support for the knowledge base article.

OBJST-1600

Name OBJMT_EVENT_PROCESSOR_FAILURE Description Object Store metering event processing lag limit exceeded.

Issue Category Manual

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies Contact Dell EMC Support for additional information.

For additional information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search support for the knowledge base article.

OBJST-1601

Name OBJMT_DELTA_LAG Description Object Store metering event processing lag limit exceeded.

Issue Category Manual

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies Contact Dell EMC Support for additional information.

For additional information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search support for the knowledge base article.

OBJST-1602

Name OBJMT_REPLICATION_FAILURE Description Object Scale replication failure detected.

Issue Category Auto, 60

Monitoring Events: Audits and Alerts 193

Notifiers objectscale-snmp-notifier Remedies Check replication settings on source bucket.

Check bucket policy settings on both source and destination buckets. For additional information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search support for the knowledge base article.

OBJST-1603

Name OBJMT_REPLICATION_NOT_PROGRESSING Description Object Scale replication is not progressing.

Issue Category Auto

Notifiers objectscale-snmp-notifier objectscale-supportassist-ese

Remedies Check the network status between source and destination Object Store. Contact Dell EMC Support for additional information. For additional information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search support for the knowledge base article.

OBJST-1604

Name ACCOUNT_HARD_QUOTA_EXCEEDED Description Hard quota on total object count or size that is exceeded for one account.

Issue Category Manual

Notifiers objectscale-snmp-notifier Remedies Increase hard count or size quota for this account or delete objects in buckets that are owned by this

account. For additional information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search support for the knowledge base article.

OBJST-1605

Name ACCOUNT_SOFT_QUOTA_EXCEEDED Description Soft quota on total object count or size that is exceeded for one account.

Issue Category Manual

Notifiers objectscale-snmp-notifier Remedies Increase soft count or size quota for this account or delete objects in buckets that are owned by this

account. For additional information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-1700

Name PRAVEGA_CONNECT_STATUS Description Pravega connection failed for at least 1 hour(default configuration).

Issue Category Auto, 60

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier

194 Monitoring Events: Audits and Alerts

Remedies Check that all Pravega pods are ready. Use DT tool listing streams or events to confirm Pravega service status. Check Pravega service logs for more details. For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-1701

Name PRAVEGA_SERVICE_STATUS Description Pravega service is unavailable.

Issue Category Auto, 60

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies Ensure that the process of object store provisioning has completed.

Check that all Pravega pods are ready. Check Pravega service logs for more details. For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-MON-4016

Name MonitoringHealth Description No data is pushed to the monitoring framework for the last 30 minutes.

Issue Category Auto, 60

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies For more information about this event, go to https://dell.com/support/objectscale and use the

SymptomID to search for the knowledge base article.

OBJST-MON-4019

Name MonitoringHealth Description No data has been pushed to the monitoring framework for the last 30 minutes.

Issue Category Auto, 60

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies Check monitoring components health (telegraf, influxdb, fluxd).

For more information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search for the knowledge base article.

OBJST-MON-4020

Name MonitoringFluxd Description Fluxd has not responded for the last 30 minutes.

Issue Category Auto, 60

Notifiers objectscale-supportassist-ese, objectscale-snmp-notifier Remedies Check Fluxd service status.

For more information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search for the knowledge base article.

Monitoring Events: Audits and Alerts 195

OBJSTORE-HC-1000

Name Pre-Update Description Preupdate health check for application.

Issue Category Auto

Notifiers objectscale-snmp-notifier Remedies Check the health status in the -app-configmap to find which checks failed.

For more information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search for the knowledge base article.

SNMPNOTI-1000

Name SNMPConnection Description SNMP connection issue.

Issue Category Auto

Notifiers -

Remedies Verify that the SNMP credentials are configured with the correct authentication values. Verify that the engineID matches with the engineID that is configured for the product in the

SNMP server. Verify that the product SNMP notifier is configured with the correct host or IP address and port. Verify the connectivity to the SNMP server, and check firewall, network routing. Verify that the SNMP v2c configuration has the correct community string.

For more information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search for the knowledge base article.

TEST TRAP

Name Test SNMP TRAP Description Test SNMP TRAP

Issue Category -

Notifiers objectscale-snmp-notifier Remedies Verify that the SNMP credentials are configured with the correct authentication values.

Verify that the engineID matches with the engineID that is configured for the product in the SNMP server.

Verify that the product SNMP notifier is configured with the correct host or IP address and port. Verify the connectivity to the SNMP server, and check firewall, network routing. Verify that the SNMP v2c configuration has the correct community string.

For additional information about this event, go to https://dell.com/support/objectscale and use the SymptomID to search for the knowledge base article.

196 Monitoring Events: Audits and Alerts

Viewing ObjectScale and object store metrics This chapter contains:

Topics:

ObjectScale metrics ObjectScale metrics in Grafana

ObjectScale metrics

Metering details within an ObjectScale instance

Various metering information is available for users on the ObjectScale instance and its object stores and other features.

ObjectScale-level metrics

When you visit the ObjectScale instance Dashboard page, the following metrics are displayed detailing current values for the ObjectScale instance:

Object Store Performance, selectable for each object store in the ObjectScale instance, shows:

Name State Read First Byte (p50) Write Last Byte (p50) Read First Byte (p99) Write Last Byte (p99) Compression ratio

ObjectScale Summary shows metering data for these areas of ObjectScale:

Health (Critical, Error, and Warning) System Data (Data Protection, Metadata, Metadata Protection, Data pending for EC, and Rate of EC per second in both

Base-2 and Base-10) Capacity Utilization (Physical Used, Available, Reserved, Total, % Full, and Days till Full (Est) in both Base-2 and Base-10) Data Management (Data Being Reclaimed, Unreclaimable Metadata, Unreclaimable User Data, Reclaimable Metadata,

Reclaimable User Data, and Capacity Reclaimed in both Base-2 and Base-10)

Object store-level metrics

When you visit an object store Dashboard page, the following metrics are displayed detailing current values for that unique object store: Latency over time for Read First Byte(p50), Write Last Byte(p50), Read First Byte(p99), and Write Last Byte(p99) Health (Critical, Error, and Warning) Physical User Data (Local Data, Replica Data, Offline Capacity Available, and Offline Capacity Recovered values in both

Base-2 and Base-10) Capacity Utilization (Physical Used, Available, Reserved, Total, % Full, and Days till Full (Est) in both Base-2 and Base-10) System Data (Data Protection, Metadata, Metadata Protection, Data pending for EC, and Rate of EC in both Base-2 and

Base-10) Logical User Data (Local Data, Replica Data, Compression Ratio, Deleted Data (24 hr), and Deleted Object Count (24 hr) in

both Base-2 and Base-10)

11

Viewing ObjectScale and object store metrics 197

Data Management (Data Being Reclaimed, Unreclaimable Metadata, Unreclaimable User Data, Reclaimable Metadata, Reclaimable User Data, and Capacity Reclaimed in both Base-2 and Base-10)

Bucket-level metrics

When you visit the Dashboard page of a bucket, the following metrics are displayed detailing current values for that unique object store: Capacity Statistics (Used Capacity Local, Used Capacity Replication, Deleted Data, TPS, Read Latency, Write Latency ) Bucket Settings (Versioning, Object Lock, Encryption) Quota Statistics (Block Size At (GB), Notification At (GB)) Object Counts (Objects > 10K, Objects 10K < 100K, Objects 100K < 1M, Objects > 1M) Event notification details (Name, Topic ARN, Events, Filter)

Account-level metrics

When you visit the Summary page of an Account, the following metrics are displayed detailing current values for that unique object store:

Alias Account ID Enabled Created On Description Groups Users Roles

Account Data

Aggregate Metrics shows total values for the selected account. These values are shown in four formats: Logical - Base-2, Logical - Base-10, Physical - Base-2, and Physical - Base-10.

Total Replica Data Total User Object Data

Hourly Metrics shows values for the selected account measured hourly. These values are shown in four formats: Logical - Base-2, Logical - Base-10, Physical - Base-2, and Physical - Base-10.

Created Object Data Deleted Object Data Created Replica Data Deleted Replica Data

ObjectScale Replication metrics

When you visit the Replication page of an object store, the following metrics are displayed detailing current values for that unique object store:

Data In (All) Data Yet To Be Replicated Failed To Replicate Objects

198 Viewing ObjectScale and object store metrics

ObjectScale metrics in Grafana

Grafana dashboards overview

The Grafana dashboards show metrics about the operation and efficiency of ObjectScale.

ObjectScale contains predefined dashboards that visualize the collected metrics. Some of the metrics are shown in the UI, on the main ObjectScale Dashboard and the object store Dashboard pages. Administrators can inspect the reported data in more detail on the Grafana dashboards. Administrators can identify developing storage and memory problems by monitoring the dashboards. The dashboards also help identify inefficiencies, and provide a way to diagnose problems.

ObjectScale metrics dashboards

NOTE: Cluster-Admin privilege is required to access ObjectScale metrics.

The predefined dashboards for an ObjectScale instance are:

Table 36. ObjectScale instance dashboards

Dashboard Description

Capacity - Overview Monitors the current and past capacity of each object store in the ObjectScale instance.

Capacity Utilization: Space Reclamation Monitors the total garbage collection and capacity reclaimed from garbage collection, current and historical.

Capacity Utilization: Used Capacity Monitors the total, used, and offline capacity data, current and historical.

Capacity Utilization: User Data Monitors the user data, current and historical.

Data Access Performance - Overview Detailed overview of the data access performance for the ObjectScale instance with views on the transaction summary, successful requests drill down, and failures drill down.

Garbage Collection: Capacity Reclaimed Monitors the amount of capacity reclaimed from garbage collection and provides a history of past capacity reclamation.

Garbage Collection: Garbage Detected Monitors the amount of garbage detected within the instance.

IAM Telemetry Details the IAM entities for each IAM account within the ObjectScale instance.

Node Rebalancing Provides an overview of any node rebalancing that has occurred, with details on the amount of data rebalanced, pending rebalancing. and the rate of rebalance (per day).

ObjectScale Overview (Default view) Monitors the ObjectScale instance.

Recovery Status Monitors the recovery status, with details on the amount of data to be recovered, recovery rate (per second), and the time to completion.

Storage Efficiency Monitors the storage efficiency.

Top Buckets Lists the top buckets in the ObjectScale instance by size and by object count.

You can access these dashboards by clicking METRICS on the Dashboard and Accounts pages within the ObjectScale instance.

Object store metrics dashboards

ObjectScale also provides metrics details for each individual object store. The predefined dashboards for an object store are:

Viewing ObjectScale and object store metrics 199

Table 37. Object store dashboards

Dashboard Description

Capacity - Overview Monitors the current and past capacity the object store.

Capacity - Overview by Disks Monitors the current and past capacity the disks.

Capacity - Overview by Nodes Monitors the current and past capacity the nodes.

Capacity Utilization: Space Reclamation Monitors the total garbage collection and capacity reclaimed from garbage collection, current and historical.

Capacity Utilization: Used Capacity Monitors the total, used, and offline capacity data, current and historical.

Capacity Utilization: User Data Monitors the user data, current and historical.

Data Access Performance - by Instance Detailed overview of the data access performance for the instance.

Data Access Performance - Overview Detailed overview of the data access performance for the ObjectScale instance with views on the transaction summary, successful requests drill down, and failures drill down.

Data Access Performance Realtime - by Instance

Provides a real time view of the data access performance by instance.

Disk Bandwidth Monitors the overall disk read/write bandwidth for the object store over the selected period of time, current and historical.

Hardware Health: All Nodes and Disks - by Nodes

Monitors the individual status of each node and their disks. Use the node_id dropdown to view details on a node.

Hardware Health: All Nodes and Disks - Overview

Monitors the status of all nodes and disks in the object store.

Hardware Health: Offline Disks Monitors the status of all offline disks in the object store.

Hardware Health: Offline Nodes Monitors the status offline nodes in the object store.

Node Rebalancing Provides an overview of any node rebalancing that has occurred, with details on the amount of data rebalanced, pending rebalancing. and the rate of rebalance (per day).

ObjectStore Overview (Default view) Monitors the object store.

Recovery - Disk Recovery Progress Monitors the current and past disk recover status.

Recovery Status Monitors the recovery status, with details on the amount of data to be recovered, recovery rate (per second), and the time to completion.

Storage Efficiency Monitors the storage efficiency.

Top Buckets Lists the top buckets in the object store by size and object count.

You can access these dashboards by clicking METRICS link of each object store.

Navigating Grafana

A Grafana dashboard is a set of one or more panels organized and arranged into one or more rows. Our ObjectScale instance ships with many pre-configured dashboards for monitoring an ObjectScale instance.

These dashboards makes it easy to quickly display properties about the functionality of the ObjectScale instance.

200 Viewing ObjectScale and object store metrics

Figure 32. Navigation details

1. Time picker dropdown. 2. Zoom out time range 3. Manual refresh button. Use to refresh all panels of the displayed dashboard. 4. Star Dashboard. Star or unstar the current Dashboard. Starred Dashboards will show up on your own Home Dashboard by

default 5. Related Dashboards. Displays the dashboards related to the currently displayed dashboard. 6. Available Dashboards. Return to the home dashboard or displays the available Dashboards. 7. Dashboard dropdown. This dropdown shows you name of the Dashboard currently displayed, and allows you to switch to a

different Dashboard. 8. Dashboard panels displaying relevant data for the dashboard.

View the Metrics dashboards for the ObjectScale instance

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Dashboard: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Dashboard. For ObjectScale Portal UI: Click the Dashboard tab. The ObjectScale Dashboard page is displayed showing an overview of the ObjectScale instance.

3. Click METRICS. The ObjectScale Overview dashboard appears.

Viewing ObjectScale and object store metrics 201

Figure 33. ObjectScale Overview dashboard

4. Use the drop-down menu in the upper left of the Metrics page to navigate between the various pre-configured ObjectScale dashboards.

View the Metrics dashboards for an object store

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Select the proper namespace from the dropdown menu.

4. Click the name of the object store.

5. Click ... if the Metrics link is not visible and then select Metrics. The ObjectStore Overview dashboard appears.

202 Viewing ObjectScale and object store metrics

Figure 34. ObjectStore Overview Dashboard

6. Use the drop-down menu in the upper left of the Metrics page to navigate between the various pre-configured object store dashboards.

Viewing ObjectScale and object store metrics 203

Troubleshooting and service procedures This chapter contains these sections on the troubleshooting and service-oriented tasks within the ObjectScale system.

Topics:

About the ObjectScale service pod Collecting troubleshooting logs About Service Procedure Operator and ObjectScale service procedures About creating a new ObjectScale object store using Helm install and a YAML config file

About the ObjectScale service pod

Working with the service pod in ObjectScale

ObjectScale utilizes a service pod to provide:

A CLI-based "working environment" that can be used to troubleshoot issues. An always-available endpoint for troubleshooting-related remote access - a "gateway" to the rest of the system. Tools and access required to troubleshoot issues, particularly advanced-level troubleshooting.

Customers and support personnel can connect into the service pod to execute troubleshooting or CLI-based maintenance tasks. The service pod is their primary environment for access - potentially the only environment they access directly while investigating an issue.

The service pod provides a Linux-based working environment with the tools and access needed to maintain the rest of the product. This would be, for example, the most common way to access kubectl and kubernetes commands, to examine system logs (from a CLI level), and as a gateway to the rest of the system.

This service pod is deployed on ObjectScale installations by default, and is expected to be running to provide remote support. The service pod also hosts the SupportAssist and remote-access functionality, although SupportAssist is not required to use the service pod.

The pod can be upgraded independently from the rest of the ObjectScale system, to provide updates as needed.

This pod also contains the utilities needed to troubleshoot the product, including standard Linux utilities (grep, awk, top, ping, ssh), Kubernetes utilities (kubectl), as well as serviceability tools and libraries developed specifically for ObjectScale.

Finally, customers and support personnel access the service pod using kubectl exec and, if remote-assist is enabled, SupportLink and SSH. Review the example in the following section to see how to access the service pod using kubectl exec.

Accessing the service pod and service tools

ObjectScale contains a service pod available in each ObjectScale instance. The service pod provides CLI access to a Linux shell environment inside the Kubernetes network.

The service pod is designed for use during troubleshooting. To access the service pod run the following command to find the name of the service pod in the ObjectScale instance:

kubectl -n get pods | grep objectscale-manager-service-pod

To access the service pod execute the kubectl exec command using similar syntax. Replace the pod name with the name of the service pod found above.

kubectl -n exec -it objectscale-manager-service-pod-5966667f5f- dxn85 -- bash

12

204 Troubleshooting and service procedures

Expected output:

objectscale-manager-service-pod-5966667f5f-dxn85: /# From here you are able to run supported service tools from the service pod. For example, the svc_log and the svc_store tools.

Service tools in this release

Name Description

svc_bucket Bucket listing/details

svc_chunk Chunk troubleshooting

svc_collect Log/data collection

svc_dt DT Troubleshooting tools (search, status)

svc_exec Run CLI commands across pods

svc_gc GC stats/troubleshooting

svc_log Log CLI - search, stats, save

svc_param CMF param list/set/revert

svc_pod Pod details, other actions

svc_request Request log utilities

svc_rest Run REST API commands

svc_store Object store details, other actions

svc_task Object store task utilities

kpi Object store workload details, performance indicators

svc_bucket list

Use this tool to list detailed information on the configured buckets.

svc_bucket list

Example usage and output

root@service-pod-objectscale:~# svc_bucket list svc_bucket (Service Tools v1.0.21.1) Started 2021-09-09 19:38:03

Owner API Encryption Versioning Lifecycle ObjectStore Bucket Name User Type Enabled Enabled Enabled -------------------------------------------------------------------------------------------------------------- ---------------------- gunjan-test bkt_random_id_1631165888 urn:osc:iam::osai6bb6025d0cc67f59:root S3 False Disabled Disabled gunjan-test test2 urn:osc:iam::osai7d14f8550f94523c:root S3 False Disabled Disabled gunjan-test test urn:osc:iam::osaiea9200f50d5420f5:root S3 False Enabled Disabled gunjan-test pravegabucket-osti289c68829c99694d urn:osc:iam::osaifeebdaedc085f9c5:root S3 False Disabled Disabled incia-obj pravegabucket-osti652c91c94e1693fa urn:osc:iam::osaifeebdaedc085f9c5:root S3 False

Troubleshooting and service procedures 205

Disabled Disabled saran-test-1 bkt_random_id_1631183477 urn:osc:iam::osaib2c1e3350dcb84ab:root S3 False Disabled Disabled saran-test-1 bkt_random_id_1631183474 urn:osc:iam::osaib2c1e3350dcb84ab:root S3 False Disabled Disabled saran-test-1 pravegabucket-ostifbf860a646eb3fcf urn:osc:iam::osaifeebdaedc085f9c5:root S3 False Disabled Disabled

Total bucket count: 8

svc_bucket info

Use this tool to display detailed information about a specific bucket.

svc_bucket info

Example usage and output

root@service-pod-objectscale:~# svc_bucket info test svc_bucket (Service Tools v1.0.21.1) Started 2021-09-09 19:38:46

Bucket ID .test

Name test Namespace osaiea9200f50d5420f5 Owner User urn:osc:iam::osaiea9200f50d5420f5:root Owner Store Name None Owner zone/Store ID urn:storageos:VirtualDataCenterData:0229acef-589b-40ea-b2e8-a39e4a968b18 Keypool Hash ID 1634da718543224a7f3529708cd7a14112f557e1e74222174709f08023e35417 Bucket Creation Date 2021-09-09 10:00:56 (1631181656136)

API Type S3 FS Access Enabled False Encryption Enabled False Versioning State Enabled Lifecycle Policy None

Bucket ACL: Type Affects Name Access Rights ------------------------------------------------------------------------ user file/dir urn:osc:iam::osaiea9200f50d5420f5:root FULL_CONTROL

kpi

Use this tool to view key performance indicators on an object store. KPIs such as the size of a workload, errors logging, and various other stats, including ingest rates and request latency.

kpi -s

Example usage and output

root@service-pod-objectscale:~# kpi -s saran-test-1 kpi (Service Tools v1.0.21.1) Started 2021-09-09 19:40:03

Object Store: saran-test-1 Time range: 2021-09-09 13:00:00 - 2021-09-09 19:40:03 Sample Time: hour

All Requests 500 Errors Timestamp GETs PUTs POSTs DELETEs HEADs Total GETs PUTs POSTs DELETEs HEADs Total

2021-09-09 13:xx 10839 16884 0 17171 17434 62328 0 0 0 0 0 0 2021-09-09 14:xx 8280 16845 0 13800 14064 52989 0 0 0 0 0 0 2021-09-09 15:xx 8930 16827 0 14351 14615 54723 0 0 0 0 0 0 2021-09-09 16:xx 8373 16716 0 13642 13902 52633 0 0 0 0 0 0 2021-09-09 17:xx 9322 14328 0 12391 12607 48648 0 0 0 0 0 0 2021-09-09 18:xx 8502 14341 0 12420 12635 47898 0 0 0 0 0 0 2021-09-09 19:xx 5318 9204 0 7499 7636 29657 0 0 0 0 0 0

206 Troubleshooting and service procedures

Totals 59564 105145 0 91274 92893 348876 0 0 0 0 0 0 Req Error %: 0.00 0.00 - 0.00 0.00 0.00

-------------------------------------------------------------------------------------------------------------- ------ Non-200 Response codes -------------------------------------------------------------------------------------------------------------- ------

Resp Code Description GETs PUTs POSTs DELETEs HEADs Total --------------------------------------------------------------------------------- 204 No Content 0 0 0 91274 0 91274 206 Partial Content 47564 0 0 0 0 47564 500 Internal Server Error 0 0 0 0 0 0

svc_request

Use this tool to view details on s3 requests from an object store.

Shown here are a few sample svc_request commands you can run.

Display a list of all requests in the last 5 minutes:

svc_request -start 5m summary

Display details for a specific request ID:

svc_request -r 0af403d4:1769f53df85:72c5:5 detail

Display all 500 errors for a specific bucket, today:

svc_request -status 500 -t PUT -start today -b ecsflex_bucket10 summary | head -50

Example usage

svc_request -start 10m -s saran-test-1 summary | more

svc_log

svc_log is the CLI interface for working with ObjectScale logs and rsyslog. It automatically locates logs for you, parallelizes search, and provides powerful filtering/ options while employing a relatively simple syntax.

svc_log

You can:

select any number of logs with simple syntax limit to logs for a single pod search logs across all pods of a particular component type, multiple components, multiple object stores, etc. do context printing (print number of lines before/after matches) print only first x lines of multi line messages collapse multi line messages to a single line include pod and/or file name

Log search is very precise, can specify specific time ranges instead of sifting through individual files.

Available search filters

Time Range Include strings (AND/OR) Exclude strings Case insensitive

Pod Selectors:

Troubleshooting and service procedures 207

Object Store/app Component(s) Log Type(s) or File name Pod Name(s) Local file(s)

root@service-pod-objectscale:/# svc_log -h usage: svc_log [-V] [-h] [-xh] {search,types,cat,list,summary,detail} ...

svc_log v1.0.4

positional arguments: {search,types,cat,list,summary,detail} sub-command help search (Default) Search logs based on component types, time ranges, filters, and more. types cat Dump the entire contents of a single log file(s) or k8s logging for one or more components. Most recent log is printed by default. list List the log(s) found on rsyslog for the specified component(s). summary Show basic information about logs available on rsyslog detail Show detailed information about each log type available on rsyslog

optional arguments: -V, --version show program's version number and exit -h, --help show this help message and exit

Global Options (Not Shown): -xh Display all help including global options (REST options, global dtquery options, etc)

Example usage and output

root@service-pod-objectscale:/# svc_log search access -a 10m > /tmp/access svc_log (Service Tools v1.0.29.0) Started 2022-01-14 17:56:10

Time range: 2022-01-14 17:46:10 - 2022-01-14 17:56:10 Filter string(s):

App or Store Name Pod Name Component File(s) ----------------------------------------------------------------- - s3 dataheadsvc-access.log*

2022-01-14 17:46:10,621 0afe075b:17e54d817bc:f29:5387 10.254.7.91:9020 10.254.4.1:42250 urn:osc:iam::osai3171ce57989a5d92:user/iam_user_testaccount2_10 - PUT osai3171ce57989a5d92 iam_user_testaccount2_10_b1_DARE e6315fb9-3b3c-4300-8111-7f8f5e7edcdf - HTTP/1.1 200 2397 131401558 - 2394 - - - 2022-01-14 17:46:11,163 0afe075b:17e54d817bc:ec8:76d3 10.254.7.91:9020 10.254.4.1:42250 urn:osc:iam::osai3171ce57989a5d92:user/iam_user_testaccount2_10 - PUT osai3171ce57989a5d92 iam_user_testaccount2_10_b8_quota_DARE 805a5797-1614-4b70-ba7c-585cdc0c30d6 - HTTP/1.1 200 540 25456359 - 538 - - - 2022-01-14 17:46:11,170 0afe075b:17e54d817bc:f29:54a0 10.254.7.91:9020 10.254.4.1:42250 urn:osc:iam::osai3171ce57989a5d92:user/iam_user_testaccount2_10 - GET osai3171ce57989a5d92 iam_user_testaccount2_10_b7_quota c1148133-7a39-48fa-9b30-4abc595a556e - HTTP/1.1 206 4 - 3412 2 bytes=23-3434 - - 2022-01-14 17:46:11,177 0afe075b:17e54d817bc:ec8:7710 10.254.7.91:9020 10.254.4.1:42250 urn:osc:iam::osai3171ce57989a5d92:user/iam_user_testaccount2_10 - GET osai3171ce57989a5d92 iam_user_testaccount2_10_b3_MetaData_quota 50e6446e-6881-4506-b09c-f3e416abad2b - HTTP/1.1 206 4 - 6750 2 bytes=14-6763 - - 2022-01-14 17:46:11,183 0afe075b:17e54d817bc:fc0:2ad3 10.254.7.91:9020 10.254.7.1:63053 urn:osc:iam::osai3171ce57989a5d92:user/iam_user_testaccount2_10 - PUT osai3171ce57989a5d92 iam_user_testaccount2_10_b3_MetaData_quota 47dd6f69-a5f4-432b-82eb-2951cab493c9 - HTTP/1.1 200 2041 116422577 - 2039 - - - ...

208 Troubleshooting and service procedures

Collecting troubleshooting logs To troubleshoot issues encountered while running ObjectScale, collecting log bundles aids in the debugging effort. Logs are critical to the root cause analysis. After encountering issues, gather these logs and log bundles.

Collect vSphere logs

Collect the WCP log bundle

Steps

1. Log in to the vSphere Client.

2. Go to Inventory view, then select the cluster that is configured for Workload Management.

3. Navigate to Supervisor Clusters.

4. Select the cluster and click on EXPORT LOGS.

The logs will be downloaded to your workstation.

Collect the NSX log bundle

Steps

1. Log in to the NSX appliance manager UI.

2. Navigate to the System tab, then Settings > Support Bundle.

3. On Request Bundle step:

a. For the Target Nodes for Log Collection, set the type to Management nodes. b. Move the nsx manager from Available to Selected. c. Set the Log age (days) to 1, to collect one day of logs. d. Set Include core files and audit logs to Yes.

4. Click START BUNDLE COLLECTION.

Collect the NCP Pod Log

Steps

1. From the Linux workstation, retrieve the name of the NSX NCP pod:

kubectl -n vmware-system-nsx get pods

NAME READY STATUS RESTARTS AGE nsx-ncp-7b6649bf9f-kdpwc 1/1 Running 15 6d9h

2. Write the pod's logs in to a file.

kubectl -n vmware-system-nsx logs nsx-ncp-7b6649bf9f-kdpwc > nsx-ncp-7b6649bf9f- kdpwc.log

3. Use scp or other method to create a copy of the log file in a location where it can be retrieved and uploaded.

4. Retrieve the file. http:// /nsx-ncp-7b6649bf9f-kdpwc.log

Troubleshooting and service procedures 209

About Service Procedure Operator and ObjectScale service procedures ObjectScale integrates into two very different environments, vSphere and OpenShift. This requires ObjectScale to support different service procedures for each of the different platforms. The ObjectScale Service Procedure Operator within the ObjectScale contains a service procedure API to manage the different service procedures for these two environments. In addition to the API, the service procedure operator also

Executes service procedure for applications that do not have their own Operator Keeps history of service procedures Keeps information for debug purposes Keeps information about different service procedures

The ObjectScale service procedure API is a complete solution to perform service procedures. API service procedure custom resource definition represents an API between user and service procedure executions. Applications watch and modify the CRD to communicate with platforms.

Depending on your ObjectScale environment, you will interact with the Service Procedure Operator and underlying Service Procedures in one of two different ways.

In vSphere deployments, use the vSphere UI. In OpenShift deployments, use kubectl.

Service procedures

In this release, ObjectScale supports the following service procedures for customers to maintain their ObjectScale deployment.

Horizontal Expansion Vertical Expansion Temporary Maintenance Mode Disk Replacement Node Replacement Upgrade ObjectScale. For more information, see About ObjectScale upgrades

NOTE: Refer to official OpenShift Container Platform upgrade procedures to:

Perform OpenShift upgrades

Perform graceful cluster shutdowns for planned outages

OpenShift upgrades must be performed within a supported version channel.

About ObjectScale capacity expansion procedures

There are two ways to increase the capacity of an object store in ObjectScale: horizontal expansions and vertical expansions.

A horizontal expansion increases the number of Storage Server Replicas, thereby increasing the number of nodes with data disks in an object store using the ObjectScale Portal UI. During the horizontal expansion ObjectScale will confirm there are enough nodes and resources on the Kubernetes environment to schedule the newly added pods. The ObjectScale Sizer tool integrated within the Edit Object Store wizard ensures that all component replica counts are increased as appropriate, according to the object store's new Storage Server Replica count.

A vertical expansion increases the number of Persistent Volumes per Storage Server Replica, thereby increasing the number of data disks per node in an object store using the ObjectScale Portal UI. During the vertical expansion ObjectScale will confirm that there is enough storage available to allocate to the newly added persistent volumes.

General guidelines or limitations of the expansion service procedures

Only one expansion operation may run at a time - either horizontal or vertical. WARNING: Parallel volume expansions of different object stores at the same time can cause node affinity

issues. If you need to expand multiple object stores, begin and complete the expansion of one object store

before beginning the expansion of the next object store.

210 Troubleshooting and service procedures

Horizontal expansions will not change the Erasure Coding (EC) scheme used by the object store. For example, if the object store was deployed with 3+3 EC, it will remain so for the life of the object store, regardless of

the number of nodes added later. Increasing the number of nodes will provide better I/O performance and increase data fragment dispersal.

During an object store expansion, the expansion will fail if certain pods are unable to start. In these cases, ObjectScale will create an alert Health > Issues tab that provides the specific details on which pod and what resource was not available. Use the details in the issue alert to resolve the underlying issue.

CAUTION: Ensure that there are sufficient Storage Capacity, Memory and CPU resources available before

you initiate an expansion procedure, otherwise the expansion may get stuck.

Ensure there is no parallel workload occurring which may consume these resources while the expansion

process is in progress. If a vertical expand procedure is initiated with fewer resources than it requires, there

is a chance for a Data Unavailability until enough resources are added and expansion process completes.

Limitations on capacity expansions in this release for the ObjectScale plugin: If you create an object store using the UI, then capacity expansions for that object store can only be enacted using the UI. If you create an object store using HELM, then capacity expansions for that object store can only be enacted using HELM

NOTE: The above restrictions come from a general limitation in this initial ObjectScale release.

Object stores created in the UI can only be modified in the UI.

Object stores created with Helm can only be modified with HELM.

Horizontally expand an object store capacity

You can expand the capacity of a previously created object store by increasing the number of Storage Server replicas in the object store.

Prerequisites

Before beginning this expansion, ensure that:

The object store is in a healthy or available status. All prior object store capacity expansions have completed.

About this task

Use these steps to expand the capacity of a previously created object store by increasing the number of Storage Server replicas in the object store.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Select the namespace from the namespace drop-down on the top right corner of the ObjectScale UI.

4. Record the information about the current capacity of the object store by:

NOTE: Each SS in the object store has the identical number of volumes. New SS replicas have the same number of

volumes as any other existing SS in the current object store.

a. In the table of object stores, click the name of object store whose capacity you want to expand. b. Click the Summary tab and locate the SS Replica Counts value in the Storage details table.

c. Close the object store view by clicking .

5. To horizontally expand the capacity of the object store.

a. In the table of object stores, reselect the object store using the checkbox to the left of the object store. b. Click EDIT to modify the object store. c. In the Edit Object Stores section, select the Storage portion of the wizard.

Troubleshooting and service procedures 211

d. Expand CUSTOMIZE(OPTIONAL) and then increase the number of Storage Server Replicas and then click Save.

Figure 35. Horizontal capacity expansion

It takes a few minutes for ObjectScale to expand the object store. Once the expansion is completed, the object store Health returns to Available.

In object store list view, you can verify that the Total Utilized Capacity increases after the horizontal expand procedure is complete.

6. Return to the Summary tab and locate the SS Replica Counts in the Storage details table for the object store.

The number of Storage Server replicas has increased.

7. Close the object store view by clicking .

Vertically expand an object store capacity

You can expand the capacity of each storage server replica by increasing the number of Volumes per Storage Server Replica within the existing object store.

Prerequisites

Before beginning this expansion, ensure that:

The object store is in a healthy or available state. All prior object store capacity expansions have completed.

212 Troubleshooting and service procedures

About this task

Use these steps to expand the number of volumes per replica of a previously created object store.

Steps

1. Log in to the ObjectScale Portal UI or VMware vSphere Client.

2. Go to the ObjectScale Object Stores: For VMware vSphere Client UI: Go to the Inventory and select the cluster that is configured for Workload Management.

Click the Configure tab. Then, scroll to the ObjectScale section and select Object Stores. For ObjectScale Portal UI: Click the Object Stores tab. The object stores page is displayed showing the current object stores in the ObjectScale instance.

3. Select the namespace from the namespace drop-down on the top right corner of the ObjectScale UI.

4. Record the information about the current capacity of the object store by:

a. In the table of object stores, click the name of object store whose volumes you want to increase. b. Click the object store Summary tab and locate the Volumes per Storage Server Replica value in the Storage details

table.

c. Close the object store view by clicking .

Total Utilized Capacity of object stores can be recorded, and it increases after vertical expansion is complete.

5. To vertically expand the object store:

a. In the table of object stores, reselect the object store using the checkbox to the left of the object store. b. Click EDIT to modify the object store. c. In the Edit Object Stores wizard, click to the Storage portion of the wizard. d. Expand CUSTOMIZE(OPTIONAL) and then increase the number of Volumes per Storage Server Replica and then

click Save.

Troubleshooting and service procedures 213

Figure 36. Vertical capacity expansion

It takes a few minutes for ObjectScale to expand the object store. Once the expansion is completed, the object store Health returns to Available.

6. Return to the object store Summary tab and locate the Volumes per Storage Server Replica in the Storage details table for the object store.

The number of PVCs has increased to accommodate the new capacity added. NOTE: During vertical expansion, each SS replica is restarted in turn. This may cause a drop in overall I/O throughput

while the pods restart and the cluster stabilizes.

7. Close the object store view by clicking .

Capacity expansion procedures using Helm

Use the helm upgrade command to modify an existing object store that is created by Helm. Do not use these procedures to modify existing object stores deployed using the ObjectScale Portal UI.

Prerequisites

Before beginning this expansion, ensure that:

The object store is in a healthy or available state. All prior object store capacity expansions have completed.

214 Troubleshooting and service procedures

About this task

Use the helm upgrade command to modify an existing object store that was created by Helm.

When horizontally expanding an object store, set the number of Storage Server replicas to a number larger than that is configured in the object store.

When vertically expanding an object store, you set the number of Volumes per Storage Server Replica to a number larger than that is configured in the object store.

CAUTION: Do not set the SS replica or Volumes per Storage Server Replica count to a number smaller than what

is configured in the object store. ObjectScale does not support a capacity shrink in this release. Using helm to

reduce the capacity of an object store has undesirable consequences and should not be attempted.

Steps

1. Check the cluster EC type, SS, and Volume values before expansion:

kubectl -n describe ecs | grep -A7 -B3'Erasure Coding'

2. Run the helm upgrade command to either horizontally or vertically expand the object store by increasing its number of SS replicas or PVCs: Horizontally expand the object store by increasing the number of SS replicas used:

helm upgrade ./ecs-cluster- .tgz -n --reuse-values --set storageServer.replicas=

NOTE: Ensure that your cluster has nodes available, as increasing storage server replicas requires additional worker

nodes in the cluster. Also, ensure you have enough drives to accommodate the additional PVCs.

Vertically expand the object store by increasing the number of volumes per SS replica used:

helm upgrade ./ecs-cluster- .tgz -n --reuse-values --set storageServer.persistence.volumesCount=

Next, after waiting for the vertical expansion to be complete, verify the new PVCs were created for each SS pod. And that the pods are up and running.

kubectl -n get pods | grep ecs-cluster-ss -w

kubectl -n get pvc | grep ecs-cluster-ss -w

3. Recheck the cluster EC type, SS, and Volume values after the expansion:

kubectl describe ecs | grep -A7 -B3'Erasure Coding'

4. Verify the ObjectScale Portal UI shows that the expansion completed successfully by reviewing the Health > Issues tab.

About maintenance modes

Taints and toleration are used to coordinate maintenance modes. A user can take a single node down for maintenance in order to repair a faulted hardware component, or they might do a sequential rolling maintenance mode (MM) to perform a software upgrade on all nodes. In the rolling MM case one node enters MM, finishes the maintenance operation, and is exited from MM and the next node is put into MM.

Temporary maintenance mode

Temporary maintenance mode (TMM) is a service procedure that is used to place a node into maintenance mode for operations like a software upgrade or other maintenance activities at the node level. While the node is in TMM, all user data remains accessible (read and write).

Troubleshooting and service procedures 215

During TMM, all object stores moves to maintenance phase until the node is taken out of TMM. Once taken out of TMM, any pending pods will start running again on the node, and the object store status returns to Avaliable.

NOTE: If the node is in successful TMM for more than 1 hour, and it is running an SS replica, recovery will begin for the

data on that node.

For ObectScale on vSphere, when you initiate Maintenance mode from the vSphere cllent UI the taint is added to the node by vSphere.

For ObectScale on OpenShift, ObjectScale users manually place a node into TMM by placing a taint on the node, after which ObjectScale operators react to the node taint and undertakes the following actions.

Once placed into TMM, all stateless pods (ReplicaSet pods) are relocated to other available nodes in the cluster automatically.

Pods that are controlled by a DaemonSet may continue to run on the node while in TMM. This includes CSI Bare-Metal node pods.

Pods that have a persistent volume claim (PVC) which is bound to a persistent volume (PV) on the node in TMM. Such pods remains in the Pending state until the node is taken out of TMM. PVCs with host local PVC, for example pods having vsan-sna-thick or vsan-direct-thick storage class remains in pending state. However other StatefulSet like influxDB, zookeeper gets migrated to other spare nodes.

If entering TMM is rejected or failed, manually return the node to an Available state. In vSphere, you do this by canceling the TMM task within the vSphere Recent Tasks section. For OpenShift ObjectScale deployments, you must manually exit the TMM by making the node schedulable. This step is not

automatic and controlled by the OpenShift administrator similar to original step used to enter TMM.

Permanent maintenance mode

Permanent maintenance mode (PMM) is a service procedure that is used to place a node into maintenance mode permanently for node removal. Permanent removal should only occur while the cluster is actively healthy. Once a taint has been placed on the node going into PMM then the ObjectScale operators will handle the taint and move resources to an available node.

In vSphere deployments, vSphere will place the taint on the node. Whereas in OpenShift deployments, you must manually set the taint on the node.

To place a node into PMM contact Dell Support and reference KB article TBD.

Maintenance mode in ObjectScale on vSphere Maintenance mode in ObjectScale on OpenShift

Maintenance mode in ObjectScale on vSphere

When performing certain maintenance tasks on ObjectScale in a vSphere environment, follow these procedures: Place a node into temporary maintenance mode in a vSphere environment Exit a node from temporary maintenance mode in a vSphere environment

Place a node into temporary maintenance mode in a vSphere environment

Temporary Maintenance Mode (TMM) allows you to place a node into maintenance mode temporarily while service or maintenance activities are conducted. The node can be exited from maintenance mode after you complete these activities.

Steps

1. Log in to VMware vSphere Client.

2. Browse to the node configured with ObjectScale which you want to place into temporary maintenance mode.

3. Right-click the node and select Maintenance Mode > Enter Maintenance Mode.

4. In the dialog box that appears select Ensure Accessibility from the dropdown, then click OK to the maintenance mode dialogue boxes that appear.

5. Ensure that the Enter Maintenance Mode process appears in the vSphere Client Recent Tasks.

216 Troubleshooting and service procedures

6. Optional: Check that the taint "node.vmware.com/drain=planned-downtime:NoSchedule" has been placed on the node:

kubectl describe node

NOTE: ObjectScale Operator puts object stores into Maintenance phase if one of object store's pods was running on

the node with temporary maintenance mode taints.

7. Retrieve the list of service procedures and locate the TMM service procedure with tmm- prefixed to the service procedure name:

kubectl -n get serviceprocedures

NAME AGE recovery-ecs-cluster-bk-bookie-0-18369f2d 32m recovery-ecs-cluster-bk-bookie-1-426c0578 26h recovery-ecs-cluster-influxdb-0-abee6329 34m recovery-ecs-cluster-influxdb-2-1d470d3e 33m recovery-ecs-cluster-zookeeper-4-c01cff83 33m recovery-objectscale-manager-influxdb-0-42e4e0a9 46m tmm-b07c77f8-397c-4a0b-a4d2-03ad3fe980c7 15m

NOTE: To obtain details about a service procedure, including its status, use:

kubectl -n describe serviceprocedures

8. Ensure that the node in the Workload Cluster is listed with (Maintenance Mode) in the vSphere Client.

9. Optional: Using kubectl, check that all evicted pods have been recreated on other nodes:

kubectl -n get pods -o wide

When a node is in maintenance mode all the stateless pods (pods without pvc/storage) & stateful set pods with 'vsan- storage protection' will get migrated from the node under TMM. All stateful set pods with a host local PVC (VSAN-SNA & VSAN-Direct) will remain in Pending state until the node exits TMM.

10. Check that all object stores are in the Maintenance phase and all the components are ready.

NOTE: For VSAN_SNA and VSAN-Direct object stores, the SS component will not be ready, if the SS pods are in

pending state.

kubectl get ecs -n -o wide

Exit a node from temporary maintenance mode in a vSphere environment

Prerequisites

You are logged into the vSphere client.

Steps

1. Browse to the node configured with ObjectScale which you want to move out of maintenance mode.

2. Right-click the node and select Maintenance Mode > Exit Maintenance Mode.

3. If applicable, click OK to the maintenance mode dialogue boxes that appear.

4. Ensure that the Exit Maintenance Mode process appears in the vSphere Client Recent Tasks.

5. Optional: Verify that the "node.vmware.com/drain=planned-downtime:NoSchedule" taint has been removed from the node:

kubectl describe node

6. Ensure that the node in the Workload Cluster is operating normally.

Troubleshooting and service procedures 217

7. Check that all of the ECS components in the ObjectScale kubernetes namespace are in the Available Status and all components are ready.

$

kubectl get ecs -n -o wide

Maintenance mode in ObjectScale on OpenShift

When performing certain maintenance tasks on ObjectScale in a OpenShift environment, follow these procedures: Enter temporary maintenance mode in an OpenShift environment Exit temporary maintenance mode in an OpenShift environment

Enter temporary maintenance mode in an OpenShift environment

Prerequisites

NOTE: When a 3+3 object store (3+3 EC, 3 SS replicas, 4 volumes/replica) is deployed on a cluster with only 3 worker

nodes, TMM will fail. The sp-operator cannot relocate pods due to anti-affinity rules. For example:

pod objectscale-portal-5f4fbd744f-5vh9m removed during Temporary Maintenance Mode is unschedulable due to 0/6 nodes are available: 1 node(s) had taint {node.dell.com/ drain: planned-downtime}, that the pod didn't tolerate, 2 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't match pod anti-affinity rules, 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.

In order to successfully complete TMM on such a deployment, PodReadyPostCheck must be set to "enabled: false" in

the sp-operator-config.yaml using

kubectl -n edit configmaps objectscale-service-procedures- config

. The operator will no longer check the status of the stateless pods (they will be pending, which is expected in this case),

allowing the TMM procedure to complete successfully.

Steps

1. Apply a taint to the node to be placed into temporary maintenance mode:

kubectl taint node node.dell.com/drain=planned-downtime:NoSchedule

2. Verify that the PHASE of the cluster now displays Maintenance.

kubectl get ecs-cluster

NAME PHASE READY COMPONENTS S3 ENDPOINT MGMT API ecs-cluster Maintenance 22/23 10.236.228.53:443 10.236.228.52:4443

The ObjectScale portal will also show the object store status as Maintenance.

3. Once the taint has been applied to a node, the Service Procedure Operator will create a TMM service procedure. Retrieve the list of service procedures and locate the TMM service procedure with tmm- prefixed to the service procedure name:

kubectl get serviceprocedures

NAME AGE recovery-ecs-cluster-bk-bookie-0-18369f2d 32m recovery-ecs-cluster-bk-bookie-1-426c0578 26h recovery-ecs-cluster-influxdb-0-abee6329 34m recovery-ecs-cluster-influxdb-2-1d470d3e 33m recovery-ecs-cluster-zookeeper-4-c01cff83 33m

218 Troubleshooting and service procedures

recovery-objectscale-manager-influxdb-0-42e4e0a9 46m tmm-a4a9b606-4126-4914-b18c-27337e841f63 15m

NOTE: To obtain details about a service procedure, including its status, use:

kubectl -n describe serviceprocedures

NOTE: Do not delete the service procedure while it is running.

4. Status retrieved periodically with the following command:

while true; do kubectl -n get serviceprocedures -o custom- columns=Name:metadata.name,Node:spec.nodeInfo.name,Type:spec.type,Time:metadata.manage dFields[0].time,Reason:status.reason,Message:status.message; echo; sleep 5; done

The service procedure will transition through various phases as it progresses. The Reason value for the TMM service procedure should progress from NotStarted, In Progress, PostCheck, and finally to Success. A reason of Success indicates the service procedure has completed without error, and the node is now in TMM.

5. Verify that only pods controlled by a DaemonSet remain running.

You may see other non-ObjectScale pods running on the node, such as metallb, openshift pods.

kubectl get pods --all-namespaces -o custom- columns=Name:metadata.name,Node:spec.nodeName,Controller:metadata.ownerReferences[*].k ind,Started:status.startTime | grep

baremetal-csi-node-56jb2 worker2.ocp4.atlantic.com DaemonSet 2021-02-19T16:11:24Z speaker-sg4ws worker2.ocp4.atlantic.com DaemonSet 2021-02-11T14:36:00Z tuned-pwphr worker2.ocp4.atlantic.com DaemonSet 2021-02-11T14:13:05Z dns-default-prbvg worker2.ocp4.atlantic.com DaemonSet 2021-02-11T14:13:05Z node-ca-tpgwk worker2.ocp4.atlantic.com DaemonSet 2021-02-11T14:13:05Z machine-config-daemon-nkdbx worker2.ocp4.atlantic.com DaemonSet 2021-02-11T14:13:05Z node-exporter-wnltt worker2.ocp4.atlantic.com DaemonSet 2021-02-11T14:13:05Z multus-m7pgs worker2.ocp4.atlantic.com DaemonSet 2021-02-11T14:13:05Z network-metrics-daemon-xp68j worker2.ocp4.atlantic.com DaemonSet 2021-02-11T14:13:05Z ovs-gm962 worker2.ocp4.atlantic.com DaemonSet 2021-02-11T14:13:05Z sdn-fcrkh worker2.ocp4.atlantic.com DaemonSet 2021-02-11T14:13:05Z

6. Additionally, any pods previously running on the TMM node, that belong to a StatefulSet, will enter the pending state. Pods that have a persistent volume claim (PVC) that is bound to a persistent volume (PV) on the node currently in TMM.

There may also be stateless pods (i.e. ReplicaSet) in the pending state. This will happen if the pods cannot be relocated due to pod anti-affinity rules. This is expected behavior.

kubectl get pods --all-namespaces=true -o custom- columns=Name:metadata.name,Node:spec.nodeName,Controller:metadata.ownerReferences[*].k ind,Status:status.phase,Started:status.startTime | grep Pending

decks-support-store-0 StatefulSet Pending objectscale-iam-atlas-1 StatefulSet Pending objs-mgr-rsyslog-4 StatefulSet Pending

7. Verify the ObjectScale Portal UI shows that the node has entered TMM by reviewing the Health > Issues tab.

Troubleshooting and service procedures 219

Results

The node is now in TMM. NOTE:

The service procedure resource only tracks the status of the Service Procedure Operator steps. The ObjectScale

Operator will also perform similar steps on pods that it controls (based on pod labels). Those steps can be tracked in the

UI under Issues/Events, or by looking at the operator logs.

Ensure that you check the Events/Issues corresponding to both ObjectScale operator and Service Procedure operator.

If the node is in TMM for more than 1 hour, and it is running an SS replica, recovery will begin automatically for the data

on that node.

Exit temporary maintenance mode in an OpenShift environment

Steps

1. Remove the taint from the node in temporary maintenance mode.

kubectl taint node node.dell.com/drain=planned-downtime:NoSchedule-

2. Verify that all nodes in the cluster are ready.

kubectl get nodes

NAME STATUS ROLES AGE VERSION master0.ocp4.atlantic.com Ready master,worker 8d v1.19.0+3b01205 master1.ocp4.atlantic.com Ready master,worker 8d v1.19.0+3b01205 master2.ocp4.atlantic.com Ready master,worker 8d v1.19.0+3b01205 worker0.ocp4.atlantic.com Ready worker 8d v1.19.0+3b01205 worker1.ocp4.atlantic.com Ready worker 8d v1.19.0+3b01205 worker2.ocp4.atlantic.com Ready worker 8d v1.19.0+3b01205 worker3.ocp4.atlantic.com Ready worker 8d v1.19.0+3b01205

3. Verify that the object store Phase returns to Available.

kubectl get ecs-cluster

4. Verify that the ObjectScale Portal UI shows that the node has returned from TMM by reviewing the Health > Events tab.

Results

All pods that were in the pending state are now running on the node as before.

Disk replacement service procedures

ObjectScale generates messages in the UI when failed PVs are found. Once failed PVs are found by ObjectScale, the disk replacement service procedure automatically begins to recover data on the failed drive. You can monitor the status of the recovery and confirm a successful recovery using the Events within the ObjectScale UI.

Shown below is the high-level overview of the disk replacement service procedure

When a disk's health becomes FAILED or SUSPECT the ObjectScale UI displays an issue in the Health > Events > Issues tab with the details of the degraded disk status.

Object Store phase during the disk replacement service procedure changes from Available to ReplacingPV Then the data recovery portion of the service procedure is initiated automatically, if applicable.

If there is available capacity (free disks) in the node, a volume is recreated once the recovery is completed using a new available disk in the system. The old disk is now waiting to be replaced (pod and object store are in a good state, and no longer impacted).

If there is no available capacity on the node (no free disks), the system will wait for the user to perform the necessary actions required by vSphere or OpenShift before continuing with the volume recreation.

The ISSUE is updated with a message that the Disk is ready for replacement

220 Troubleshooting and service procedures

The User replaces the disk. The ISSUE is updated with a message that the Disk has been successfully replaced.

Automatic disk replacement service procedure - vSphere

About this task

For ObjectScale on vSphere, the disk replacement procedure is implemented within ObjectScale and automatically handles disk failures. This procedure details the process to locate failed Persistent Volumes and then successfully replace the PV.

Steps

1. Log in to the VMware vSphere Client.

2. Navigate to the ObjectScale object stores.

a. Go to the Inventory view. b. Select the cluster configured for Workload Management. c. Click the Configure tab and then scroll and select Object Stores listed under the ObjectScale section.

The list of Object Stores that the user is authorized to view is displayed.

3. Locate the object store with ReplacingPV status.

When a PV fails, the state and health of the object store(s) containing that PV will go into ReplacingPV and the disk replacement service procedure begins.

4. Monitor the status of the process at Monitor > ObjectScale > Health > Events tab to view the system generated events for the disk replacement process. If the system contains an available spare drive, the service procedure will progress to completion. Once the process has

completed, the object store(s) will return to the Started State and Available Health. If there are no available spare drives or otherwise insufficient capacity, the disk replacement service procedure generates

a warning events for Not enough capacity as it attempts to recreate the PVCs on the failed PV.

To resolve, put the "disk.dell.com/replacement": "ready" annotation to the failed PVC immediately before you are ready to reinsert physical drive. Once you place this annotation, the old unhealthy PVCs are deleted automatically.

5. Physically replace the failed drive or insert a new drive into the system. To locate the failed drive in the system enclosures,

For additional information on replacing a capacity device within vSphere, refer to https://docs.vmware.com/en/VMware- vSphere/7.0/com.vmware.vsphere.vsan-monitoring.doc/GUID-4E3390C1-6C50-49E5-AEB6-C9BC037979A1.html.

a. Go to Inventory > Configure > Disk Management. b. Select the node on which to search for failed disks and then select VIEW DISKS. c. Search for the failed disk(s):

For vSAN Disks, navigate to Disk Groups and look for a failed vSAN disk. For vSAN Direct Disks, navigate to vSAN Direct Disks to look for a failed VSAN Direct disk.

6. After replacing the failed disk or adding a new disk, go to Configure > vSAN > Disk Management and click CLAIM UNUSED DISKS.

The Claim Unused Disks wizard appears.

7. Complete the necessary steps in the wizard to either claim the disk as either for vSAN Direct or not for vSAN direct and click Create.

8. After the spare drive(s) are added to vSphere and is available, the disk replacement service procedure will progress to completion.

Automatic disk replacement service procedure - OpenShift

About this task

For ObjectScale on OpenShift, the disk replacement procedure is implemented within ObjectScale and automatically handles disk failures. This procedure details the process to locate failed Persistent Volumes and then successfully replace the PV.

Troubleshooting and service procedures 221

Steps

1. Go to the ObjectScale instance and click on Object Stores. The object stores details page appears. When a PV fails, the state and health of the object store(s) containing that PV will go into ReplacingPV and the disk replacement service procedure begins.

2. Monitor the status of the process at Monitor > ObjectScale > Health > Events tab to view the system generated events for the disk replacement process. If the system contains an available spare drive, the service procedure will progress to completion. Once the process has

completed, the object store(s) will return to the Started State and Available Health. If there are no available spare drives or otherwise insufficient capacity, the disk replacement service procedure generates

a warning events for Not enough capacity as it attempts to recreate the PVCs on the failed PV. Complete step 3 to complete the replacement if this occurs.

3. After getting an event that Reason: DriveReadyForRemoval and have a new disk available, initiate the physical replacement in OpenShift by placing a replacement=ready annotation on the failed/suspect disk.

a. Confirm the disk is in Released status.

kubectl get drives | grep

b. Place the replacement=ready annotation on the failed/suspect disk.

annotate drives.csi-baremetal.dell.com replacement=ready

c. Confirm that the disk is now in Removed status.

kubectl get drives | grep

d. Confirm that the ISSUE has been updated with Reason: DriveReadyForPhysicalRemoval.

CAUTION: Do not physical replace the disk until the above WARNING event is displayed under the

respective ISSUE.

The disk LED is blinking. If you are unable to identify the disk to replace, you will need to determine another way to identify the disk manually or visually, by using additional information located in the associated ISSUE events.

4. Remove and replace the failed drive with the new, clean drive. Afterwards, the ISSUE in ObjectScale Portal UI will be auto-cleared by being set to Normal severity. Once the event Reason: "DriveSuccessfullyRemoved" occurs and you have inserted a new drive into the node, the disk replacement service procedure has completed successfully and no further actions is required.

Node replacement service procedures for vSphere and OpenShift

Node replacement service procedure for vSphere

To replace a node in a vSphere system running ObjectScale, complete these steps.

Steps

1. Prepare the node to be replaced. If the node is healthy, follow Place a node into temporary maintenance mode in a vSphere environment

to prepare the node to be replaced by placing the node into TMM. Then, you must remove the host from the inventory (https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vcenterhost.doc/GUID- C88D843A-DB67-4888-9C36-8B72335EF3F8.html).

If the node is failed, you must remove the host from the inventory (https://docs.vmware.com/en/VMware- vSphere/7.0/com.vmware.vsphere.vcenterhost.doc/GUID-C88D843A-DB67-4888-9C36-8B72335EF3F8.html).

2. Physically replace the node hardware. Then, transfer the disks to the replacement node. Install and add the node back to the cluster.

When replacing a failed node, ensure that you follow the VMware vSphere procedure to add a new ESXi host into the cluster.

3. If you were unable to reuse the disks from the node, complete these additional steps after installing the replaced node:

a. Remove the PVCs of the stateful pods that are stuck.

222 Troubleshooting and service procedures

b. Delete any ObjectScale stateful pods in Pending status.

4. Ensure that the object stores have all returned to Available status.

kubectl -n get ecs

5. Ensure that the pods have all returned to Running status.

kubectl -n get pods

Node replacement service procedure for OpenShift

To replace a node in a OpenShift cluster running ObjectScale, complete these steps.

Prerequisites

Ensure that the replacement node has the same name and IP address as the node being replaced. If the replacement process takes longer than 1 hour (which is likely), recovery will begin to run for the data on the replaced

node. However, it should stop once the node is replaced and operational.

Steps

1. Prepare the node for removal: If the node is healthy, follow Enter temporary maintenance mode in an OpenShift environment to prepare the node to be

replaced by placing the node into TMM. If the node is in a failed

state, follow https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.6/ html/replacing_nodes/openshift_container_storage_deployed_using_local_storage_devices#replacing-a-failed-node-on- bare-metal-user-provisioned-infrastructure_rhocs a. From the service node, run the following command to mark the node as unscheduable:

kubectl cordon b. Remove the pods in Terminating state:

kubectl get pods -A -o wide | grep-i | awk '{if ($4 == "Terminating") system ("kubectl -n " $1 " delete pods " $2 " --grace-period=0 " " --force ")}'

c. Drain the node by evacuating the pods from the node: NOTE: To list objects that will be evacuated without actually preforming the operation, include the --dry- run=client parameter with the command below.

kubectl drain --force --delete-local-data --ignore-daemonsets

2. Remove the node from the cluster:

kubectl delete node

3. Physically remove and replace the failed node hardware. As you do so, ensure that

you move all of the drives from the failed node into the new compute node, and install and join it back to the OpenShift cluster, following the steps outlined in the OpenShift documentation.

the new node satisfies the requirements listed in the "Deployment pre-requisites for ObjectScale on OpenShift" of the Dell EMC ObjectScale Installation Guide for OpenShift.

All of the PVC bindings remain, all the stateful pods will start on the new node.

4. Ensure that the new node has been added to the cluster, and all nodes are ready. For example:

kubectl get nodes

NAME STATUS ROLES AGE VERSION master0.ocp4.atlantic.com Ready master 15d v1.19.0+e49167a master1.ocp4.atlantic.com Ready master 15d v1.19.0+e49167a

Troubleshooting and service procedures 223

master2.ocp4.atlantic.com Ready master 15d v1.19.0+e49167a worker0.ocp4.atlantic.com Ready worker 15d v1.19.0+e49167a worker1.ocp4.atlantic.com Ready worker 46m v1.19.0+e49167a worker2.ocp4.atlantic.com Ready worker 15d v1.19.0+e49167a

5. Verify that the node has been recognized by CSI and appears in the baremetal node list. For example:

kubectl get csibmnodes

NAME UUID ADDRESSES csibmnode-4f19a3e9-9c9b-40a8-... 4f19a3e9-9c9b-40a8-... {"Hostname":"master0.ocp4.atlantic.com","InternalIP":"10.236.224.60"} csibmnode-a0dba2b4-5eab-4c34-... a0dba2b4-5eab-4c34-... {"Hostname":"worker0.ocp4.atlantic.com","InternalIP":"10.236.224.66"} csibmnode-bb7dcedc-139b-4d8f-... bb7dcedc-139b-4d8f-... {"Hostname":"master2.ocp4.atlantic.com","InternalIP":"10.236.224.64"} csibmnode-bdb9f0b8-f52d-4aaf-... bdb9f0b8-f52d-4aaf-... {"Hostname":"worker1.ocp4.atlantic.com","InternalIP":"10.236.224.68"} csibmnode-de3eebf0-dfcd-41e9-... de3eebf0-dfcd-41e9-... {"Hostname":"worker3.ocp4.atlantic.com","InternalIP":"10.236.224.72"} csibmnode-e820eea6-3145-4fb8-... e820eea6-3145-4fb8-... {"Hostname":"master1.ocp4.atlantic.com","InternalIP":"10.236.224.62"} csibmnode-f21e396b-2d91-43d5-... f21e396b-2d91-43d5-... {"Hostname":"worker2.ocp4.atlantic.com","InternalIP":"10.236.224.70"}

6. Verify the cluster is available. For example:

kubectl get ecs

NAME PHASE READY COMPONENTS S3 ENDPOINT MGMT API ecs-cluster Available 23/23 10.236.228.53:443 10.236.228.52:4443

7. Verify other features and components, these include:

S3 I/O ObjectScale Portal kubectl command output All pods, including those previously in the pending state, are now running Pod restarts have not occurred or increased

8. Optional: Monitor the automatic recovery of non-SS pods:

After the new node has joined the cluster, and the pods are running, the Service Procedure Operator may initiate recovery procedures for certain pods. Recovery will be started for bookie, influxdb and zookeeper pods, if any were running on the removed node.

kubectl get serviceprocedures -A -o custom- columns=Name:metadata.name,Node:spec.nodeInfo.name,Type:spec.type,Time:metadata.manage dFields[0].time,Reason:status.reason,Message:status.message

Troubleshooting Service Procedures

View service procedure status

You can view the status of the service procedure with:

kubectl get serviceprocedures

Permanent Maintenance Mode service procedure status

When the PMM service procedure is in progress: Phase is changed to "Maintenance" status.conditions.maintenance.type = "in progress"OR status.conditions.maintenance.type = "completed" (All migrations are completed but taints still on a node)

Once the PMM service procedure is completed:

224 Troubleshooting and service procedures

Phase is changed to "Available" status.conditions.maintenance.type = "success"

If the PMM service procedure failed: Alert is sent status.conditions.maintenance.type = "failed"

Temporary Maintenance Mode service procedure status

When the TMM service procedure is in progress: Phase is changed to "Maintenance" status.conditions.maintenance.type = "in progress"

Once the TMM service procedure is completed: Phase is changed to "Available" status.conditions.replacement.type = "success"

If the TMM service procedure failed: Alert is sent status.conditions.replacement.type = "failed"

Disk Replacement service procedure status

When the replacement is in progress: Phase is changed to "PVReplacing" status.conditions.replacement.type = "in progress"OR status.conditions.replacement.type = "pvc deleted"(PVCs were deleted and we need to delete ss one more time)

Once the replacement is completed: Phase is changed to "Available" status.conditions.replacement.type = "success"

If the replacement failed: Alert is sent status.conditions.replacement.type = "failed"

Upgrade service procedure status

When the upgrade service procedure is in progress: Phase is changed to "Upgrade" status.conditions.upgrading.type = "in progress"

Once the upgrade service procedure is completed: Phase is changed to "Available" status.conditions.upgrading.type = "success"

If the upgrade service procedure failed:

NOTE: If an upgrade fails and you need to rollback to a previous version, you must to contact Dell EMC support.

Alert is sent status.conditions.upgrading.type = "failed" status.conditions.upgrading.reason is not ""

Vertical expansion service procedure status

When the vertical expansion service procedure is in progress: Phase is changed to "Expansion" status.conditions.expansion.type = "in progress"

Once the vertical expansion service procedure is completed: Phase is changed to "Available" status.conditions.expansion.type = "success"

If the vertical expansion service procedure failed: Alert is sent

Troubleshooting and service procedures 225

status.conditions.expansion.type = "failed"

Horizontal expansion service procedure status

When the horizontal expansion service procedure is in progress: Phase is changed to "Expansion" status.conditions.expansion.type = "in progress", status.conditions.updatingTopology.type = "in progress" When new pod is Running, status.conditions.expansion.type = "succeed" and phase is changed to "UpdatingTopology"

Once the horizontal expansion service procedure is completed: Phase is changed to "Available" status.conditions.expansion.type = "success" status.conditions.updatingTopology.type = "success"

If the horizontal expansion service procedure failed: Alert is sent status.conditions.expansion.type = "failed"

or

status.conditions.updatingTopology.type = "failed"

Service Procedure states

Retrieve details on the service procedures using:

kubectl describe serviceprocedures

apiVersion: ecs.dellemc.com/v1beta3 kind: ServiceProcedure metadata: label: app: ecs-release-name # in case for a specific service procedure spec: type: Enum(PermanentMaintenanceMode) # A type of service procedure diskInfo: name: name of replacing disk # Contains K8s PVC Name uuid: UUID of the replacing disk (if applicable) # On Openshift it's resolving from the Volume CRD nodeInfo: name: name of replacing node # Contains K8s Node Name uuid: UUID of the tainted node # Contains K8s Node UID status: reason: Enum(In Progress, Success, Failed, Recovering, Rollback, Abort) # current actual state message: # message what is going on for rightnow.

The Service Procedure custom resource (CR) can have the following state in the status.reason field:

1. Created - New SP CR recently created by the SP Operator when service procedure event was detected. It should have filled spec.type and spec.diskInfo or spec.nodeInfo fields.

2. NotStarted - A state of the SP with passed pre-checks. Ready for further processing. 3. Recovering - A state applicable only for components where recovery scripts are available. SP CR is in Recovering state

after Created and before In Progress. 4. In Progress - A state of the processing SP CR. In general, this occurs after the Created state. 5. PostCheck - A state of the SP after main processing. The SP operator runs post checks until SP's post-check fails (if one

of handling pods in the Failed state) or succeed (is all handling pods in a Running'state). 6. Failed - Terminated state of the SP CR in case of any failure during SP processing or failed post-check. 7. Rejected - Terminated state of the SP CR if one of pre-checks failed and further processing is not allowed. 8. Success - Terminated state of the succeed SP CR.

226 Troubleshooting and service procedures

About creating a new ObjectScale object store using Helm install and a YAML config file ObjectScale on OpenShift supports creating an object store using helm. Once you create an object store with helm, this object store can only be managed by helm. Object stores created with Helm can be monitored but cannot be edited, upgraded, or deleted with the UI.

For the ObjectScale initial release, there are three supported object store erasure coding (EC) deployment schemes: 3+3, 12+4, and 10+2. The table below shows the values used to define each of the object store types.

EC Scheme Data Blocks Code Blocks Nodes Volumes/Node Deployment Method

3+3 3 3 3 4 UI/Helm

12+4 12 4 5 5 UI/Helm

10+2 10 2 7 3 Helm

Object stores using the 3+3 and 12+4 EC schemes should be primarily created using the ObjectScale Portal UI. When an object store is created from the UI, the number of nodes is chosen based on the EC scheme. The logic/rules within the Create an Object Store wizard (sizer) automatically sets the number of pod replicas deployed for each component, based on the number of nodes.

Alternatively, 3+3 and 12+4 object stores may be also created using Helm. Helm installation is required for object stores using the 10+2 EC scheme. Since there is no sizer logic for an object store created with Helm, all relevant values, including the replica count for each object store component, must be set manually as part of the install command.

Dell EMC ObjectScale recommends using the simplified alternative creation method shown below to running the long-hand, complicated helm install command. A YAML config file may be employed instead. The YAML file contains all of the parameters required to deploy the object store, and alleviates the need to specify them with command-line parameters.

For details about using helm to create new object stores or to use helm to manage certain aspects of the ObjectScale instance, see https://dell.com/support/objectscale.

Manually create an Object Store using helm install for ObjectScale

Certain customers may choose to manage ObjectScale object stores using helm commands, either manually or through automation.

Refer to "Create a new ObjectScale object store using Helm install and a YAML config file", found in a KB article at https:// dell.com/support/objectscale. Follow these topics to create an object store in ObjectScale using the helm install commands.

NOTE: Any object store created using the helm install can only be managed using helm and cannot be deleted, expanded,

or upgraded using the ObjectScale Portal UI. Likewise, object stores created using the ObjectScale Portal UI, cannot be

managed using helm.

Troubleshooting and service procedures 227

Miscellaneous management tasks for ObjectScale on vSphere

This appendix contains:

Topics:

Create vSphere namespace and users for ObjectScale Create a custom vSAN SNA striped policy for the object store namespace

Create vSphere namespace and users for ObjectScale After deploying ObjectScale, vSphere Admins can create the vSphere end-user accounts to access the storage within ObjectScale. You can sort these users into groups for easier management. Then, you can create and configure a new end-user namespace in vSphere for these end-users. Before beginning, review the following high-level overview of the steps that will be needed to create the vSphere resources.

Log in to the vSphere UI as local admin. As the Admin user, create one or more end-user- namespace(s) where you can deploy end-user object stores

As the local admin user, create additional vSphere users and groups. Then, assign the admin role to one of the users so that it can manage the cluster as admin while logged in as a user. Assign the necessary roles and permissions to the end-users and groups to manage access to ObjectScale resources. As the Admin user, assign RBAC roles to the users so that they can manage the specified namespace and the object stores.

Add vCenter users for ObjectScale

Users listed on the Users tab in the vSphere Client are internal to vCenter Single Sign-On and belong to the .local domain.

Steps

1. Log in to the vSphere Client as the vSphere administrator.

2. Navigate to the vCenter Single Sign-On user configuration UI.

a. From the Home menu, select Administration. b. Under Single Sign On section, click Users and Groups.

3. On the Users tab, select the appropriate Domain and click ADD.

4. In the Add User wizard, type the necessary information for the new user:

a. Type a username and password for the new user.

You cannot change the user name after you create a user. The password must meet the password policy requirements for the system.

b. Provide additional details (first and last name, email address, and description) for the new user.

5. Click Add.

Configure permissions for vSphere end-users

Add permissions to the namespace to allow end-user access to the namespace and then configure the vSphere user permissions.

About this task

To configure permissions for the vSphere end-users of ObjectScale, complete the following steps.

A

228 Miscellaneous management tasks for ObjectScale on vSphere

Steps

1. Click on the newly created namespace, navigate to the Summary tab, and click Add Permissions.

2. Configure the permissions at the Kubernetes namespace-level for the end-users.

a. Set the Identity source. b. Search for the User/Group that will have access to the namespace. c. Define the Role for the selected User/Group.

For View-only users, set the role to Can view. For Edit users, set the role to Can edit.

3. Configure the permissions at vCenter-level. To provide edit permissions to a user created with the Can edit role:

a. Navigate to Administration > Access Control > Roles b. Create new role by cloning the existing Workload Storage Manager role.

By default, this role provides users with basic view-only permissions, so you will need to edit the cloned role.

c. Name the cloned role ObjectScaleEditRole and click on OK.

d. Select the ObjectScaleEditRole role to select the Host > Inventory and click SAVE.

To provide view-only permissions to a user: a. Navigate to Administration > Access Control > Roles b. Create new role by cloning the existing Workload Storage Manager role. This role provides users with basic

view-only permissions c. Name the cloned role ObjectScaleBasicRole and click on OK.

Create and configure a new ObjectScale end-user namespace

Within the vSphere Workload Cluster associated with ObjectScale, create a new end-user namespace.

Prerequisites

Create users or groups for all DevOps engineers who will access the namespace. ObjectScale storage policies for persistent storage that vSphere Pods and pods inside a Tanzu Kubernetes cluster will use. Required privileges:

Namespaces.Modify cluster-wide configuration Namespaces.Modify namespace configuration

Steps

1. Log in to the vSphere Client as the vSphere administrator.

2. From the vSphere Client home menu, select Workload Cluster.

3. Click Namespaces and click New Namespace.

4. Create the new namespace:

a. Select the Workload Cluster where you want to place the namespace. b. Enter a name for the namespace.

The name must be in a DNS-compliant format.

c. Enter a description, and click Create.

The namespace is created on the Supervisor Cluster.

5. Set permissions so that end users can access the namespace.

a. From the Permissions pane, select Add Permissions. b. Select the identity source, the ObjectScale or other vSphere user(s) or the group, and set the role from the dropdown

menu, and click OK.

Repeat this step for all the vSphere users created or otherwise identified to access ObjectScale.

6. Set persistent storage to the namespace.

a. From the Storage pane, select Add Storage. b. Select the ObjectScale storage policies and click OK.

Miscellaneous management tasks for ObjectScale on vSphere 229

Create a custom vSAN SNA striped policy for the object store namespace

About this task

ObjectScale recommends that vSphere users who plan to use vSAN SNA, splits/stripes the volumes across the drives in the node, create a custom vSAN SNA striped policy for the object store namespace. With this custom policy, you will reduce the stripe width to two less than the number of drives in the node. This custom policy allows ObjectScale to rebuild the pods on the node (via recovery process) even when 1 or 2 disks are offline.

Steps

1. Go to the Policies and Profiles > VM Storage Profiles and click Create.

The Create VM Storage Policy wizard appears.

2. Enter the policy name and description, and click Next.

3. On the Policy structure page:

a. Under Host based services, disable host based rules. b. Under Datastore specific rules, enable rules for "vSAN" storage.

4. On the Availability portion of the VSAN page, ensure the following settings are applied: Site disaster tolerance value is set to None - standard cluster. Failures to tolerate value is set to No data redundancy with host affinity.

5. On the Storage rules portion of the VSAN page, ensure the following settings are applied or leave these values as default (No Preference): Encryption services value is set to No encrytion. Space efficiency value is set to No space efficiency. Storage tier value is set to Hybrid.

6. On the Advanced Policy Rules portion of the VSAN page, modify the number of disk stripes for the policy. Use the default values for all of the other advanced policy rule settings. :

If you have 6 or more VSAN capacity disks per node: stripe width should be set to 2 less than the number of capacity disks per node (max. 12). For example, if you have 8 VSAN capacity disks per node, then the stripe width should be 6.

Otherwise, leave stripe width set to 1

7. On the Storage compatibility page, review the list of datastores that match this policy.

8. On the Review and finish page, review the storage policy settings and click Finish.

To change any settings, click Back.

Next, edit the object store namespace(s) to include the new custom VSAN storage policy.

9. Log in to the vSphere Client as the vSphere administrator.

10. Go to Inventory and select vCenter Cluster.

11. Click Namespaces to expand the tree view and then select the object store namespace.

12. On the Summary

Manualsnet FAQs

If you want to find out how the V1.0 Dell works, you can view and download the Dell ObjectScale V1.0 Storage Administration Guide on the Manualsnet website.

Yes, we have the Administration Guide for Dell V1.0 as well as other Dell manuals. All you need to do is to use our search bar and find the user manual that you are looking for.

The Administration Guide should include all the details that are needed to use a Dell V1.0. Full manuals and user guide PDFs can be downloaded from Manualsnet.com.

The best way to navigate the Dell ObjectScale V1.0 Storage Administration Guide is by checking the Table of Contents at the top of the page where available. This allows you to navigate a manual by jumping to the section you are looking for.

This Dell ObjectScale V1.0 Storage Administration Guide consists of sections like Table of Contents, to name a few. For easier navigation, use the Table of Contents in the upper left corner.

You can download Dell ObjectScale V1.0 Storage Administration Guide free of charge simply by clicking the “download” button in the upper right corner of any manuals page. This feature allows you to download any manual in a couple of seconds and is generally in PDF format. You can also save a manual for later by adding it to your saved documents in the user profile.

To be able to print Dell ObjectScale V1.0 Storage Administration Guide, simply download the document to your computer. Once downloaded, open the PDF file and print the Dell ObjectScale V1.0 Storage Administration Guide as you would any other document. This can usually be achieved by clicking on “File” and then “Print” from the menu bar.