Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions docs/fileset-catalog-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,13 @@ Gravitino Fileset catalog index includes the following chapters:

## Fileset Catalog with Cloud Storage

Apart from the above, you can also refer to the following topics to manage and access cloud storage like S3, GCS, ADLS, and OSS:
Each cloud backend has its own page with a runnable end-to-end example, from installing the bundle
jar to reading the fileset with Spark, the Hadoop shell, Python and pandas:

- [Using Fileset catalog to manage S3](./fileset-catalog-with-s3.md).
- [Using Fileset catalog to manage GCS](./fileset-catalog-with-gcs.md).
- [Using Fileset catalog to manage ADLS](./fileset-catalog-with-adls.md).
- [Using Fileset catalog to manage OSS](./fileset-catalog-with-oss.md).
- [Using Fileset catalog to manage Amazon S3](./fileset-catalog-with-s3.md).
- [Using Fileset catalog to manage Google Cloud Storage](./fileset-catalog-with-gcs.md).
- [Using Fileset catalog to manage Azure Data Lake Storage](./fileset-catalog-with-adls.md).
- [Using Fileset catalog to manage Alibaba Cloud OSS](./fileset-catalog-with-oss.md).
- [Using Fileset catalog to manage Tencent Cloud COS](./fileset-catalog-with-cos.md).

More storage options will be added soon. Stay tuned!
581 changes: 269 additions & 312 deletions docs/fileset-catalog-with-adls.md

Large diffs are not rendered by default.

596 changes: 292 additions & 304 deletions docs/fileset-catalog-with-cos.md

Large diffs are not rendered by default.

509 changes: 229 additions & 280 deletions docs/fileset-catalog-with-gcs.md

Large diffs are not rendered by default.

561 changes: 257 additions & 304 deletions docs/fileset-catalog-with-oss.md

Large diffs are not rendered by default.

572 changes: 265 additions & 307 deletions docs/fileset-catalog-with-s3.md

Large diffs are not rendered by default.

46 changes: 26 additions & 20 deletions docs/fileset-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,15 @@ license: "This software is licensed under the Apache License version 2."

## Introduction

Fileset catalog is a fileset catalog that using Hadoop Compatible File System (HCFS) to manage
the storage location of the fileset. It supports the local filesystem and HDFS.
Gravitino supports [S3](fileset-catalog-with-s3.md), [GCS](fileset-catalog-with-gcs.md),
[OSS](fileset-catalog-with-oss.md) and [Azure Blob Storage](fileset-catalog-with-adls.md) through Fileset catalog.
Gravitino also supports [Tencent Cloud COS](fileset-catalog-with-cos.md).
The fileset catalog manages the storage location of a fileset through a Hadoop Compatible File
System (HCFS). It supports the local filesystem and HDFS out of the box, and Amazon S3, Google Cloud
Storage, Azure Data Lake Storage, Alibaba Cloud OSS and Tencent Cloud COS once the matching bundle
jar is on the classpath.

The rest of this document will use HDFS or local file as an example to illustrate how to use the Fileset catalog.
For S3, GCS, OSS, Azure Blob Storage and COS, the configuration is similar to HDFS,
refer to the corresponding document for more details.
This page is the shared reference: the properties every backend accepts, how they are inherited from
catalog to schema to fileset, and how to plug in a custom filesystem. It uses HDFS and the local
filesystem in its examples. For a runnable end-to-end example on a cloud backend, follow the page for
that backend listed under [Fileset Catalog with Cloud Storage](#fileset-catalog-with-cloud-storage).

Note that Gravitino uses Hadoop 3 dependencies to build Fileset catalog. Theoretically, it should be
compatible with both Hadoop 2.x and 3.x, since Gravitino doesn't leverage any new features in
Expand All @@ -26,7 +26,7 @@ Hadoop 3. If there's any compatibility issue, create an [issue](https://github.c

### Catalog Properties

Besides the [common catalog properties](./gravitino-server-config.md#catalog-properties-configuration),
Besides the [common catalog properties](./gravitino-server-config.md#catalog-properties),
the Fileset catalog has the following properties:

| Property Name | Description | Default Value | Required |
Expand Down Expand Up @@ -75,15 +75,21 @@ The Gravitino Fileset extends the following properties in the `xxx-site.xml`:

### Fileset Catalog with Cloud Storage

In the current implementation, the fileset uses the HDFS protocol to access its location. If users use S3, GCS, OSS,
Azure Blob Storage or Tencent Cloud COS, they can also configure the `config.resources` to specify custom configuration
files.
A fileset always reaches its location through the HDFS protocol, so a cloud backend is a matter of
putting the right bundle jar on the classpath and setting the credential properties for that
backend. Each backend has its own page with a runnable end-to-end example.

- For S3, refer to [Fileset-catalog-with-s3](./fileset-catalog-with-s3.md) for more details.
- For GCS, refer to [Fileset-catalog-with-gcs](./fileset-catalog-with-gcs.md) for more details.
- For OSS, refer to [Fileset-catalog-with-oss](./fileset-catalog-with-oss.md) for more details.
- For Azure Blob Storage, refer to [Fileset-catalog-with-adls](./fileset-catalog-with-adls.md) for more details.
- For Tencent Cloud COS, refer to [Fileset-catalog-with-cos](./fileset-catalog-with-cos.md) for more details.
| Storage backend | Bundle jar | Location scheme | Backend properties |
|------------------------------------------------------------|-----------------------------|-----------------|-----------------------------------------------------------------------------|
| [Amazon S3](./fileset-catalog-with-s3.md) | `gravitino-aws-bundle` | `s3a://` | `s3-endpoint`, `s3-access-key-id`, `s3-secret-access-key` |
| [Google Cloud Storage](./fileset-catalog-with-gcs.md) | `gravitino-gcp-bundle` | `gs://` | `gcs-service-account-file` |
| [Azure Data Lake Storage](./fileset-catalog-with-adls.md) | `gravitino-azure-bundle` | `abfss://` | `azure-storage-account-name`, `azure-storage-account-key` |
| [Alibaba Cloud OSS](./fileset-catalog-with-oss.md) | `gravitino-aliyun-bundle` | `oss://` | `oss-endpoint`, `oss-access-key-id`, `oss-secret-access-key` |
| [Tencent Cloud COS](./fileset-catalog-with-cos.md) | `gravitino-tencent-bundle` | `cosn://` | `cos-region`, `cos-access-key-id`, `cos-secret-access-key`, `cos-endpoint` |

A catalog may hold locations in more than one backend at the same time, as long as every bundle jar
involved is on the classpath. Cloud backends also accept `config.resources` to pass custom
configuration files to the underlying filesystem client.

### Implement a Custom HCFS File System Fileset

Expand Down Expand Up @@ -134,7 +140,7 @@ value, and the priority mechanism is the same as authentication.

### Catalog Operations

Refer to [Catalog operations](./manage-fileset-metadata-using-gravitino.md#catalog-operations) for more details.
Refer to [Catalog operations](./manage-catalogs-and-schemas.md#catalog-operations) for more details.

## Schema

Expand All @@ -160,15 +166,15 @@ properties:

### Schema Operations

Refer to [Schema operation](./manage-fileset-metadata-using-gravitino.md#schema-operations) for more details.
Refer to [Schema operations](./manage-catalogs-and-schemas.md#schema-operations) for more details.

:::note
During schema creation or deletion, Gravitino automatically creates or removes the corresponding filesystem directories
for the schema locations.
This behavior is skipped in either of these cases:

1. When the catalog property `disable-filesystem-ops` is set to `true`
2. When the location contains [placeholders](./manage-fileset-metadata-using-gravitino.md#placeholder)
2. When the location contains [placeholders](./filesets.md#storage-locations)
:::

## Fileset
Expand Down
2 changes: 1 addition & 1 deletion docs/hive-catalog-with-cloud-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ Once all configurations have been correctly set, restart the Hive cluster to app

## Create Tables or Databases with S3 Storage Using Gravitino

Assuming you have already set up a Hive catalog with Gravitino, you can proceed to create tables or databases using S3 storage. For more information on catalog operations, refer to [Catalog operations](./manage-fileset-metadata-using-gravitino.md#catalog-operations)
Assuming you have already set up a Hive catalog with Gravitino, you can proceed to create tables or databases using S3 storage. For more information on catalog operations, refer to [Catalog operations](./manage-catalogs-and-schemas.md#catalog-operations)

### Example: Creating a Database with S3 Storage

Expand Down
31 changes: 19 additions & 12 deletions docs/how-to-use-gvfs.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ To configure the Gravitino client, use properties prefixed with `fs.gravitino.cl

:::note
When users work with a multi-cluster fileset catalog, they can configure separate sets of properties for the base paths
of the different clusters. [Manage filesets across multiple clusters](./manage-fileset-metadata-using-gravitino.md#manage-filesets-across-multiple-clusters)
of the different clusters, using the `fs.path.config.<name>` properties described above.

For example, a complex catalog structure might look like this:

Expand Down Expand Up @@ -112,11 +112,14 @@ The plain `fs.path.config.<name>` entry specifies the base path of the filesyste

**Note:** Invalid configuration properties will result in exceptions. Please see [Gravitino Java client configurations](./how-to-use-gravitino-client.md#java-client-configuration) for more support client configuration.

Apart from the above properties, to access fileset like S3, GCS, OSS and custom fileset, extra properties are needed; see
[S3 GVFS Java client configurations](./fileset-catalog-with-s3.md#access-the-fileset-with-the-gvfs-java-client),
[GCS GVFS Java client configurations](./fileset-catalog-with-gcs.md#access-the-fileset-with-the-gvfs-java-client),
[OSS GVFS Java client configurations](./fileset-catalog-with-oss.md#access-the-fileset-with-the-gvfs-java-client)
and [Azure Blob Storage GVFS Java client configurations](./fileset-catalog-with-adls.md#access-the-fileset-with-the-gvfs-java-client) for more details.
A fileset backed by cloud storage needs the credential properties of that backend on top of the
properties above, and the matching bundle jar on the classpath. See
[Amazon S3](./fileset-catalog-with-s3.md#amazon-s3-properties),
[Google Cloud Storage](./fileset-catalog-with-gcs.md#google-cloud-storage-properties),
[Azure Data Lake Storage](./fileset-catalog-with-adls.md#azure-data-lake-storage-properties),
[Alibaba Cloud OSS](./fileset-catalog-with-oss.md#alibaba-cloud-oss-properties) and
[Tencent Cloud COS](./fileset-catalog-with-cos.md#tencent-cloud-cos-properties) for the property
names and a runnable example for each backend.

#### Custom Fileset

Expand Down Expand Up @@ -439,7 +442,7 @@ To configure the Gravitino client, use properties prefixed with `gvfs_gravitino_

:::note
When users work with a multi-cluster fileset catalog, they can configure separate sets of properties for the base paths
of the different clusters. [Manage filesets across multiple clusters](./manage-fileset-metadata-using-gravitino.md#manage-filesets-across-multiple-clusters)
of the different clusters, using the `fs.path.config.<name>` properties described above.
Comment thread
yuqi1129 marked this conversation as resolved.
Outdated

For example, a complex catalog structure might look like this:

Expand Down Expand Up @@ -471,12 +474,16 @@ options = {
The plain `fs_path_config_<name>` entry specifies the base path of the filesystem. Any additional key under the same prefix (`fs_path_config_<name>_<config_key>`) is treated as a location-scoped configuration (for example, `config.resource` for HDFS) and is forwarded directly to the underlying filesystem client.
:::

#### Configurations for S3, GCS, OSS and Azure Blob Storage Fileset
#### Configurations for Cloud Storage Filesets

Please see the cloud-storage-specific configurations [GCS GVFS Python client configurations](./fileset-catalog-with-gcs.md#access-the-fileset-with-the-gvfs-python-client),
[S3 GVFS Python client configurations](./fileset-catalog-with-s3.md#access-the-fileset-with-the-gvfs-python-client),
[OSS GVFS Python client configurations](./fileset-catalog-with-oss.md#access-the-fileset-with-the-gvfs-python-client)
and [Azure Blob Storage GVFS Python client configurations](./fileset-catalog-with-adls.md#access-the-fileset-with-the-gvfs-python-client) for more details.
A fileset backed by cloud storage needs the credential properties of that backend, spelled with
underscores rather than hyphens. See
[Amazon S3](./fileset-catalog-with-s3.md#amazon-s3-properties),
[Google Cloud Storage](./fileset-catalog-with-gcs.md#google-cloud-storage-properties),
[Azure Data Lake Storage](./fileset-catalog-with-adls.md#azure-data-lake-storage-properties),
[Alibaba Cloud OSS](./fileset-catalog-with-oss.md#alibaba-cloud-oss-properties) and
[Tencent Cloud COS](./fileset-catalog-with-cos.md#tencent-cloud-cos-properties) for the property
names and a runnable example for each backend.
Comment thread
yuqi1129 marked this conversation as resolved.

:::note
Gravitino python client does not support [customized file systems](fileset-catalog.md#implement-a-custom-hcfs-file-system-fileset) defined by users due to the limit of `fsspec` library.
Expand Down
Loading