diff --git a/docs/fileset-catalog-index.md b/docs/fileset-catalog-index.md index 4a94f6391ea..a03e2ecbd73 100644 --- a/docs/fileset-catalog-index.md +++ b/docs/fileset-catalog-index.md @@ -16,11 +16,14 @@ Gravitino Fileset catalog index includes the following chapters: ## Fileset Catalog with Cloud Storage -Apart from the above, you can also refer to the following topics to manage and access cloud storage like S3, GCS, ADLS, and OSS: +Each cloud backend has its own page with a runnable end-to-end example covering catalog setup and +Java/Hadoop data access. The S3, GCS, ADLS, and OSS pages also cover Python and pandas; COS +currently has no Python data-plane implementation: -- [Using Fileset catalog to manage S3](./fileset-catalog-with-s3.md). -- [Using Fileset catalog to manage GCS](./fileset-catalog-with-gcs.md). -- [Using Fileset catalog to manage ADLS](./fileset-catalog-with-adls.md). -- [Using Fileset catalog to manage OSS](./fileset-catalog-with-oss.md). +- [Using Fileset catalog to manage Amazon S3](./fileset-catalog-with-s3.md). +- [Using Fileset catalog to manage Google Cloud Storage](./fileset-catalog-with-gcs.md). +- [Using Fileset catalog to manage Azure Data Lake Storage](./fileset-catalog-with-adls.md). +- [Using Fileset catalog to manage Alibaba Cloud OSS](./fileset-catalog-with-oss.md). +- [Using Fileset catalog to manage Tencent Cloud COS](./fileset-catalog-with-cos.md). More storage options will be added soon. Stay tuned! diff --git a/docs/fileset-catalog-with-adls.md b/docs/fileset-catalog-with-adls.md index 1331b72f948..be3fa2f83e5 100644 --- a/docs/fileset-catalog-with-adls.md +++ b/docs/fileset-catalog-with-adls.md @@ -1,62 +1,65 @@ --- title: "Fileset Catalog with ADLS" slug: "/fileset-catalog-with-adls" -date: 2025-01-03 -keyword: "Fileset catalog ADLS" +keyword: "Fileset catalog ADLS Azure Blob Storage" license: "This software is licensed under the Apache License version 2." --- ## Introduction -This document describes how to configure a Fileset catalog with ADLS (aka. Azure Blob Storage (ABS), or Azure Data Lake Storage (v2)). +This page shows how to store fileset data in Azure Data Lake Storage while Gravitino manages the metadata, +and how to read and write that data through the Gravitino Virtual File System (GVFS). -## Prerequisites +Everything on this page is specific to Azure Data Lake Storage. The fileset model itself, the properties shared by +every storage backend, and the way properties are inherited from catalog to schema to fileset are +described in [Fileset Catalog](./fileset-catalog.md). + +The examples run in order and use the same names throughout: metalake `metalake`, catalog +`adls_catalog`, schema `adls_schema`, fileset `example_fileset`, and `http://localhost:8090` as the +server URL. Replace them with your own values. -To set up a Fileset catalog with ADLS, follow these steps: +## Prerequisites 1. Download the [`gravitino-azure-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-azure-bundle) file. -2. Place the downloaded file into the Gravitino Fileset catalog classpath at `${GRAVITINO_HOME}/catalogs/fileset/libs/`. -3. Start the Gravitino server by running the following command: +2. Place it in the fileset catalog classpath at `${GRAVITINO_HOME}/catalogs/fileset/libs/`. +3. Start the Gravitino server: ```bash -$ ${GRAVITINO_HOME}/bin/gravitino-server.sh start +${GRAVITINO_HOME}/bin/gravitino-server.sh start ``` -Once the server is up and running, you can proceed to configure the Fileset catalog with ADLS. In the rest of this document we will use `http://localhost:8090` as the Gravitino server URL, replace with your actual server URL. +The catalog automatically loads the Azure Data Lake Storage filesystem provider once the bundle jar is on the +classpath. The deprecated `filesystem-providers` and `default-filesystem-provider` catalog +properties do not need to be set. -## ADLS Catalog Configuration +## Azure Data Lake Storage Properties -### ADLS Fileset Catalog Configuration +These properties are needed in addition to the shared +[catalog properties](./fileset-catalog.md#catalog-properties). The same values are also needed by +the GVFS clients, so they are listed together here — note that the Python client spells them with +underscores while the catalog and the Java client use hyphens. -Apart from configurations mentioned in [fileset-catalog-catalog-configuration](./fileset-catalog.md#catalog-properties), the following properties are required to configure a Fileset catalog with ADLS: - -| Configuration item | Description | Default value | Required | -|-------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|----------| -| `filesystem-providers` | (deprecated) The file system providers to add. Set it to `abs` if it's a Azure Blob Storage fileset, or a comma separated string that contains `abs` like `oss,abs,s3` to support multiple kinds of fileset including `abs`. | (none) | Yes | -| `default-filesystem-provider` | (deprecated) The name default filesystem providers of this Fileset catalog if users do not specify the scheme in the URI. Default value is `builtin-local`, for Azure Blob Storage, if we set this value, we can omit the prefix 'abfss://' in the location. | `builtin-local` | No | -| `azure-storage-account-name ` | The account name of Azure Blob Storage. | (none) | Yes | -| `azure-storage-account-key` | The account key of Azure Blob Storage. | (none) | Yes | -| `credential-providers` | The credential provider types, separated by comma, possible value can be `adls-token`, `azure-account-key`. As the default authentication type is using account name and account key as the above, this configuration can enable credential vending provided by Gravitino server and client will no longer need to provide authentication information like account_name/account_key to access ADLS by GVFS. Once it's set, more configuration items are needed to make it works, see [adls-credential-vending](security/credential-vending.md#adls-credentials) | (none) | No | +| Catalog and Java client | Python client | Description | Required | +|------------------------------|------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------| +| `azure-storage-account-name` | `azure_storage_account_name` | Account name of the Azure Blob Storage. | Yes | +| `azure-storage-account-key` | `azure_storage_account_key` | Account key of the Azure Blob Storage. | Yes | +| `credential-providers` | (n/a) | The credential provider types, separated by comma. Possible values are `adls-token`, `azure-account-key`. Setting it enables credential vending, so clients no longer need the credentials above. See [credential vending](./security/credential-vending.md#adls) for the extra properties each provider takes. | No | :::note -`default-filesystem-provider` and `filesystem-providers` are deprecated. The fileset catalog automatically loads filesystem providers on the classpath, including buildin filesystem provider and cloud providers when the corresponding bundle jar is present (for example, `gravitino-azure-bundle`). +Azure Data Lake Storage is also known as Azure Blob Storage (ABS). The location uses the `abfss://` +scheme. ::: -### Schema Configuration - -Refer to [Schema configurations](./fileset-catalog.md#schema-properties) for more details. - -### Fileset Configuration +Schema and fileset properties are documented on the shared page: see +[schema properties](./fileset-catalog.md#schema-properties) and +[fileset properties](./fileset-catalog.md#fileset-properties). -Refer to [Fileset configurations](./fileset-catalog.md#fileset-properties) for more details. +A fileset catalog stores its data under `location`, which for Azure Data Lake Storage looks like +`abfss://container@account-name.dfs.core.windows.net/root`. ## Create the Catalog, Schema, and Fileset -This section demonstrates how to create the Fileset catalog with ADLS in Gravitino, with a complete example. - -### Step 1: Create a Fileset Catalog with ADLS - -First, you need to create a Fileset catalog with ADLS. The following example shows how to create a Fileset catalog with ADLS: +### Step 1: Create the catalog @@ -64,13 +67,13 @@ First, you need to create a Fileset catalog with ADLS. The following example sho ```shell curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{ - "name": "example_catalog", + "name": "adls_catalog", "type": "FILESET", - "comment": "This is a ADLS fileset catalog", + "comment": "A fileset catalog backed by Azure Data Lake Storage", "properties": { - "location": "abfss://container@account-name.dfs.core.windows.net/path", - "azure-storage-account-name": "The account name of the Azure Blob Storage", - "azure-storage-account-key": "The account key of the Azure Blob Storage" + "location": "abfss://container@account-name.dfs.core.windows.net/root", + "azure-storage-account-name": "account_name", + "azure-storage-account-key": "account_key" } }' http://localhost:8090/api/metalakes/metalake/catalogs ``` @@ -84,44 +87,42 @@ GravitinoClient gravitinoClient = GravitinoClient .withMetalake("metalake") .build(); -Map adlsProperties = ImmutableMap.builder() - .put("location", "abfss://container@account-name.dfs.core.windows.net/path") - .put("azure-storage-account-name", "azure storage account name") - .put("azure-storage-account-key", "azure storage account key") +Map catalogProperties = ImmutableMap.builder() + .put("location", "abfss://container@account-name.dfs.core.windows.net/root") + .put("azure-storage-account-name", "account_name") + .put("azure-storage-account-key", "account_key") .build(); -Catalog adlsCatalog = gravitinoClient.createCatalog("example_catalog", - Type.FILESET, - "This is a ADLS fileset catalog", - adlsProperties); -// ... - +Catalog catalog = gravitinoClient.createCatalog("adls_catalog", + Catalog.Type.FILESET, + "A fileset catalog backed by Azure Data Lake Storage", + catalogProperties); ``` ```python -gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake") -adls_properties = { - "location": "abfss://container@account-name.dfs.core.windows.net/path", - "azure-storage-account-name": "azure storage account name", - "azure-storage-account-key": "azure storage account key" +gravitino_client: GravitinoClient = GravitinoClient( + uri="http://localhost:8090", metalake_name="metalake") + +catalog_properties = { + "location": "abfss://container@account-name.dfs.core.windows.net/root", + "azure-storage-account-name": "account_name", + "azure-storage-account-key": "account_key", } -adls_properties = gravitino_client.create_catalog(name="example_catalog", - catalog_type=Catalog.Type.FILESET, - provider=None, - comment="This is a ADLS fileset catalog", - properties=adls_properties) +catalog = gravitino_client.create_catalog(name="adls_catalog", + catalog_type=Catalog.Type.FILESET, + provider=None, + comment="A fileset catalog backed by Azure Data Lake Storage", + properties=catalog_properties) ``` -### Step 2: Create a Schema - -Once the catalog is created, you can create a schema. The following example shows how to create a schema: +### Step 2: Create the schema @@ -129,49 +130,44 @@ Once the catalog is created, you can create a schema. The following example show ```shell curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{ - "name": "test_schema", - "comment": "This is a ADLS schema", + "name": "adls_schema", + "comment": "A schema in the Azure Data Lake Storage fileset catalog", "properties": { - "location": "abfss://container@account-name.dfs.core.windows.net/path" + "location": "abfss://container@account-name.dfs.core.windows.net/root/schema" } -}' http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas +}' http://localhost:8090/api/metalakes/metalake/catalogs/adls_catalog/schemas ``` ```java -Catalog catalog = gravitinoClient.loadCatalog("test_catalog"); - +Catalog catalog = gravitinoClient.loadCatalog("adls_catalog"); SupportsSchemas supportsSchemas = catalog.asSchemas(); Map schemaProperties = ImmutableMap.builder() - .put("location", "abfss://container@account-name.dfs.core.windows.net/path") + .put("location", "abfss://container@account-name.dfs.core.windows.net/root/schema") .build(); -Schema schema = supportsSchemas.createSchema("test_schema", - "This is a ADLS schema", - schemaProperties -); -// ... + +Schema schema = supportsSchemas.createSchema("adls_schema", + "A schema in the Azure Data Lake Storage fileset catalog", + schemaProperties); ``` ```python -gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake") -catalog: Catalog = gravitino_client.load_catalog(name="test_catalog") -catalog.as_schemas().create_schema(name="test_schema", - comment="This is a ADLS schema", - properties={"location": "abfss://container@account-name.dfs.core.windows.net/path"}) +catalog: Catalog = gravitino_client.load_catalog(name="adls_catalog") +catalog.as_schemas().create_schema(name="adls_schema", + comment="A schema in the Azure Data Lake Storage fileset catalog", + properties={"location": "abfss://container@account-name.dfs.core.windows.net/root/schema"}) ``` -### Step 3: Create a Fileset - -After creating the schema, you can create a fileset. The following example shows how to create a fileset: +### Step 3: Create the fileset @@ -182,346 +178,365 @@ curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ "name": "example_fileset", "comment": "This is an example fileset", "type": "MANAGED", - "storageLocation": "abfss://container@account-name.dfs.core.windows.net/path/example_fileset", + "storageLocation": "abfss://container@account-name.dfs.core.windows.net/root/schema/example_fileset", "properties": { "k1": "v1" } -}' http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas/test_schema/filesets +}' http://localhost:8090/api/metalakes/metalake/catalogs/adls_catalog/schemas/adls_schema/filesets ``` ```java -GravitinoClient gravitinoClient = GravitinoClient - .builder("http://localhost:8090") - .withMetalake("metalake") - .build(); - -Catalog catalog = gravitinoClient.loadCatalog("test_catalog"); +Catalog catalog = gravitinoClient.loadCatalog("adls_catalog"); FilesetCatalog filesetCatalog = catalog.asFilesetCatalog(); -Map propertiesMap = ImmutableMap.builder() - .put("k1", "v1") - .build(); +Map filesetProperties = ImmutableMap.builder() + .put("k1", "v1") + .build(); filesetCatalog.createFileset( - NameIdentifier.of("test_schema", "example_fileset"), + NameIdentifier.of("adls_schema", "example_fileset"), "This is an example fileset", Fileset.Type.MANAGED, - "abfss://container@account-name.dfs.core.windows.net/path/example_fileset", - propertiesMap, -); + "abfss://container@account-name.dfs.core.windows.net/root/schema/example_fileset", + filesetProperties); ``` ```python -gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake") - -catalog: Catalog = gravitino_client.load_catalog(name="test_catalog") -catalog.as_fileset_catalog().create_fileset(ident=NameIdentifier.of("test_schema", "example_fileset"), - type=Fileset.Type.MANAGED, - comment="This is an example fileset", - storage_location="abfss://container@account-name.dfs.core.windows.net/path/example_fileset", - properties={"k1": "v1"}) +catalog: Catalog = gravitino_client.load_catalog(name="adls_catalog") +catalog.as_fileset_catalog().create_fileset( + ident=NameIdentifier.of("adls_schema", "example_fileset"), + type=Fileset.Type.MANAGED, + comment="This is an example fileset", + storage_location="abfss://container@account-name.dfs.core.windows.net/root/schema/example_fileset", + properties={"k1": "v1"}) ``` -## Access a Fileset with ADLS +The fileset is now addressable as +`gvfs://fileset/adls_catalog/adls_schema/example_fileset` from any GVFS client. -### Access the Fileset with the GVFS Java Client +## Access the Fileset -To access fileset with Azure Blob Storage(ADLS) using the GVFS Java client, based on the [basic GVFS configurations](./how-to-use-gvfs.md#configuration-1), you need to add the following configurations: +### Java client jars -| Configuration item | Description | Default value | Required | -|------------------------------|-----------------------------------------|---------------|----------| -| `azure-storage-account-name` | The account name of Azure Blob Storage. | (none) | Yes | -| `azure-storage-account-key` | The account key of Azure Blob Storage. | (none) | Yes | +Every Java or Hadoop-based client needs `gravitino-filesystem-hadoop3-runtime`, which is published +on Maven Central, plus the Azure Data Lake Storage filesystem implementation. Only the latter +differs by environment: -:::note -If the catalog has enabled [credential vending](security/credential-vending.md), the properties above can be omitted. More details can be found in [Fileset with credential vending](#fileset-with-credential-vending). -::: +| Environment | Jar providing the Azure Data Lake Storage filesystem | +|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| No Hadoop installed | [`gravitino-azure-bundle`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-azure-bundle), a fat jar bundling the Azure Data Lake Storage filesystem implementation and its dependencies | +| Hadoop already present | `hadoop-azure-${hadoop-version}.jar`, `azure-storage-7.0.1.jar` and `wildfly-openssl-1.0.7.Final.jar`, shipped with Hadoop under `${HADOOP_HOME}/share/hadoop/tools/lib` | -```java -Configuration conf = new Configuration(); -conf.set("fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs"); -conf.set("fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem"); -conf.set("fs.gravitino.server.uri", "http://localhost:8090"); -conf.set("fs.gravitino.client.metalake", "test_metalake"); -conf.set("azure-storage-account-name", "account_name_of_adls"); -conf.set("azure-storage-account-key", "account_key_of_adls"); -Path filesetPath = new Path("gvfs://fileset/test_catalog/test_schema/test_fileset/new_dir"); -FileSystem fs = filesetPath.getFileSystem(conf); -fs.mkdirs(filesetPath); -... -``` +The artifacts in full: -Similar to Spark configurations, you need to add ADLS (bundle) jars to the classpath according to your environment. +- [`gravitino-azure-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-azure-bundle): + a "fat" jar that includes the `gravitino-azure` functionality together with every dependency it needs, + such as `hadoop-azure` and the packages it needs to reach ADLS. Use it when the environment has no pre-existing Hadoop setup. +- [`gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-filesystem-hadoop3-runtime): + a "fat" jar that bundles the Gravitino virtual filesystem client and already includes the + `gravitino-azure` functionality. Java and Hadoop-based clients require it to access Gravitino + filesets. +- `hadoop-azure-${hadoop-version}.jar`, `azure-storage-7.0.1.jar` and + `wildfly-openssl-1.0.7.Final.jar`: the standard Hadoop dependencies for Azure Data Lake Storage + access, shipped with Hadoop under `${HADOOP_HOME}/share/hadoop/tools/lib`. Supply them yourself + when running inside an existing Hadoop environment. +- [`gravitino-azure-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-azure): + a "thin" jar carrying only the Azure integration code. It is already contained in both jars above, + so it is not needed as a direct dependency unless you prefer to manage all Hadoop and Azure + dependencies yourself. -If you want to custom your hadoop version or there is already a hadoop version in your project, you can add the following dependencies to your `pom.xml`: +```xml + + + org.apache.gravitino + gravitino-azure-bundle + ${GRAVITINO_VERSION} + + + org.apache.gravitino + gravitino-filesystem-hadoop3-runtime + ${GRAVITINO_VERSION} + +``` ```xml - - org.apache.hadoop - hadoop-common - ${HADOOP_VERSION} - - - - org.apache.hadoop - hadoop-azure - ${HADOOP_VERSION} - - - - org.apache.gravitino - gravitino-filesystem-hadoop3-runtime - ${GRAVITINO_VERSION} - + + + org.apache.hadoop + hadoop-common + ${HADOOP_VERSION} + + + org.apache.hadoop + hadoop-azure + ${HADOOP_VERSION} + + + org.apache.gravitino + gravitino-filesystem-hadoop3-runtime + ${GRAVITINO_VERSION} + ``` :::note -The `gravitino-azure` JAR is no longer required, as it is now included in the `gravitino-filesystem-hadoop3-runtime` JAR. +The thin `gravitino-azure` jar is not needed. Its functionality is already included in both +`gravitino-azure-bundle` and `gravitino-filesystem-hadoop3-runtime`. ::: -Or use the bundle jar with Hadoop environment if there is no Hadoop environment: +### GVFS Java client -```xml - - org.apache.gravitino - gravitino-azure-bundle - ${GRAVITINO_VERSION} - - - - org.apache.gravitino - gravitino-filesystem-hadoop3-runtime - ${GRAVITINO_VERSION} - -``` +On top of the [base GVFS configuration](./how-to-use-gvfs.md#configuration), set the Azure Data Lake Storage +properties from the table above. -### Access the Fileset with Spark +```java +Configuration conf = new Configuration(); +conf.set("fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs"); +conf.set("fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem"); +conf.set("fs.gravitino.server.uri", "http://localhost:8090"); +conf.set("fs.gravitino.client.metalake", "metalake"); +conf.set("azure-storage-account-name", "account_name"); +conf.set("azure-storage-account-key", "account_key"); -The following code snippet shows how to use **PySpark 3.5.0 with Hadoop environment(Hadoop 3.3.4)** to access the fileset: +Path filesetPath = new Path("gvfs://fileset/adls_catalog/adls_schema/example_fileset/new_dir"); +FileSystem fs = filesetPath.getFileSystem(conf); +fs.mkdirs(filesetPath); +``` -Before running the following code, you need to install required packages: +### Apache Spark + +The example below uses PySpark 3.5.0 in an environment that already has Hadoop 3.3.4. ```bash pip install pyspark==3.5.0 pip install apache-gravitino==${GRAVITINO_VERSION} ``` -Then you can run the following code: ```python -from pyspark.sql import SparkSession import os +from pyspark.sql import SparkSession -gravitino_url = "http://localhost:8090" -metalake_name = "test" +# On JDK 17, also add: +# --conf "spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" +# --conf "spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" +os.environ["PYSPARK_SUBMIT_ARGS"] = ( + "--jars /path/to/gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar," + "/path/to/hadoop-azure-3.3.4.jar," + "/path/to/azure-storage-7.0.1.jar," + "/path/to/wildfly-openssl-1.0.7.Final.jar " + "--master local[1] pyspark-shell" +) -catalog_name = "your_adls_catalog" -schema_name = "your_adls_schema" -fileset_name = "your_adls_fileset" -# JDK8 as follows, JDK17 will be slightly different, you need to add '--conf \"spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED\" --conf \"spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED\"' to the submit args. -os.environ["PYSPARK_SUBMIT_ARGS"] = "--jars /path/to/gravitino-filesystem-hadoop3-runtime-{gravitino-version}.jar,/path/to/hadoop-azure-3.3.4.jar,/path/to/azure-storage-7.0.1.jar,/path/to/wildfly-openssl-1.0.7.Final.jar --master local[1] pyspark-shell" -spark = SparkSession.builder - .appName("adls_fileset_test") +spark = (SparkSession.builder + .appName("adls_fileset") .config("spark.hadoop.fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs") .config("spark.hadoop.fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem") .config("spark.hadoop.fs.gravitino.server.uri", "http://localhost:8090") - .config("spark.hadoop.fs.gravitino.client.metalake", "test") - .config("spark.hadoop.azure-storage-account-name", "azure_account_name") - .config("spark.hadoop.azure-storage-account-key", "azure_account_key") - .config("spark.hadoop.fs.azure.skipUserGroupMetadataDuringInitialization", "true") + .config("spark.hadoop.fs.gravitino.client.metalake", "metalake") + .config("spark.hadoop.azure-storage-account-name", "account_name") + .config("spark.hadoop.azure-storage-account-key", "account_key") .config("spark.driver.memory", "2g") .config("spark.driver.port", "2048") - .getOrCreate() + .getOrCreate()) data = [("Alice", 25), ("Bob", 30), ("Cathy", 45)] -columns = ["Name", "Age"] -spark_df = spark.createDataFrame(data, schema=columns) -gvfs_path = f"gvfs://fileset/{catalog_name}/{schema_name}/{fileset_name}/people" - -spark_df.coalesce(1).write - .mode("overwrite") - .option("header", "true") - .csv(gvfs_path) -``` - -If your Spark **without Hadoop environment**, you can use the following code snippet to access the fileset: - -```python -## Replace the following code snippet with the above code snippet with the same environment variables +spark_df = spark.createDataFrame(data, schema=["Name", "Age"]) +gvfs_path = "gvfs://fileset/adls_catalog/adls_schema/example_fileset/people" -os.environ["PYSPARK_SUBMIT_ARGS"] = "--jars /path/to/gravitino-azure-bundle-{gravitino-version}.jar,/path/to/gravitino-filesystem-hadoop3-runtime-{gravitino-version}.jar --master local[1] pyspark-shell" +spark_df.coalesce(1).write.mode("overwrite").option("header", "true").csv(gvfs_path) ``` -If Spark can't start with the above configuration (no Hadoop environment available and use bundle jar), you can try to set the jars to the classpath directly: -```python -jars_path = ( - "/path/to/gravitino-azure-bundle-{gravitino-version}.jar:" - "/path/to/gravitino-filesystem-hadoop3-runtime-{gravitino-version}.jar" -) +If Spark runs without a Hadoop environment, only the jar list changes: +```python os.environ["PYSPARK_SUBMIT_ARGS"] = ( - f'--conf "spark.driver.extraClassPath={jars_path}" ' - f'--conf "spark.executor.extraClassPath={jars_path}" ' - '--master local[1] pyspark-shell' + "--jars /path/to/gravitino-azure-bundle-${gravitino-version}.jar," + "/path/to/gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar " + "--master local[1] pyspark-shell" ) ``` -- [`gravitino-azure-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-azure-bundle): A "fat" JAR that includes `gravitino-azure` functionality and all necessary dependencies like `hadoop-azure` (3.3.1) and other packages needed to access ADLS. Use this if your Spark environment doesn't have a pre-existing Hadoop setup. -- [`gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-filesystem-hadoop3-runtime): A "fat" JAR that bundles Gravitino's virtual filesystem client and includes the functionality of `gravitino-azure`. It is required for accessing Gravitino filesets. -- `hadoop-azure-3.3.4.jar`, `azure-storage-7.0.1.jar`, and `wildfly-openssl-1.0.7.Final.jar`: Standard Hadoop dependencies for ADLS access. If you are running in an existing Hadoop environment, you need to provide these JARs. They are typically located in the `${HADOOP_HOME}/share/hadoop/tools/lib` directory. -- [`gravitino-azure-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-azure): A "thin" JAR that only provides the Azure integration code. Its functionality is already included in the `gravitino-azure-bundle` and `gravitino-filesystem-hadoop3-runtime` JARs, so you do not need to add it as a direct dependency unless you want to manage all Hadoop and Azure dependencies manually. - -Please choose the correct jar according to your environment. - :::note -In some Spark versions, a Hadoop environment is necessary for the driver, adding the bundle jars with '--jars' may not work. If this is the case, you should add the jars to the spark CLASSPATH directly. +Some Spark versions need a Hadoop environment in the driver and do not pick up filesystem +implementations passed with `--jars`. If that happens, add the jars to the Spark classpath directly. ::: -### Access a Fileset Using the Hadoop Fs Command +### Hadoop fs command -The following are examples of how to use the `hadoop fs` command to access the fileset in Hadoop 3.1.3: - -1. Adding the following contents to the `${HADOOP_HOME}/etc/hadoop/core-site.xml` file: +1. Add the following to `${HADOOP_HOME}/etc/hadoop/core-site.xml`: ```xml - - fs.AbstractFileSystem.gvfs.impl - org.apache.gravitino.filesystem.hadoop.Gvfs - - - - fs.gvfs.impl - org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem - - - - fs.gravitino.server.uri - http://localhost:8090 - - - - fs.gravitino.client.metalake - test - - - - azure-storage-account-name - account_name - - - azure-storage-account-key - account_key - + + fs.AbstractFileSystem.gvfs.impl + org.apache.gravitino.filesystem.hadoop.Gvfs + + + fs.gvfs.impl + org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem + + + fs.gravitino.server.uri + http://localhost:8090 + + + fs.gravitino.client.metalake + metalake + + + azure-storage-account-name + account_name + + + azure-storage-account-key + account_key + ``` -2. Add the necessary jars to the Hadoop classpath. +2. Add these jars to the Hadoop classpath: -For ADLS, you need to add `gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar` and `hadoop-azure-${hadoop-version}.jar` located at `${HADOOP_HOME}/share/hadoop/tools/lib/` to the Hadoop classpath. + - `gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`, from Maven Central. + - `hadoop-azure-${hadoop-version}.jar`, `azure-storage-7.0.1.jar` and `wildfly-openssl-1.0.7.Final.jar`, shipped with Hadoop under `${HADOOP_HOME}/share/hadoop/tools/lib`. -3. Run the following command to access the fileset: +3. Access the fileset: ```shell -./${HADOOP_HOME}/bin/hadoop dfs -ls gvfs://fileset/adls_catalog/adls_schema/adls_fileset -./${HADOOP_HOME}/bin/hadoop dfs -put /path/to/local/file gvfs://fileset/adls_catalog/adls_schema/adls_fileset +${HADOOP_HOME}/bin/hadoop fs -ls gvfs://fileset/adls_catalog/adls_schema/example_fileset +${HADOOP_HOME}/bin/hadoop fs -put /path/to/local/file gvfs://fileset/adls_catalog/adls_schema/example_fileset ``` -### Access the Fileset with the GVFS Python Client - -To access fileset with Azure Blob storage (ADLS) using the GVFS Python client, apart from [basic GVFS configurations](./how-to-use-gvfs.md#configuration-1), you need to add the following configurations: - -| Configuration item | Description | Default value | Required | -|------------------------------|----------------------------------------|---------------|----------| -| `azure_storage_account_name` | The account name of Azure Blob Storage | (none) | Yes | -| `azure_storage_account_key` | The account key of Azure Blob Storage | (none) | Yes | - -:::note -If the catalog has enabled [credential vending](security/credential-vending.md), the properties above can be omitted. -::: - -Please install the `gravitino` package before running the following code: +### GVFS Python client ```bash pip install apache-gravitino==${GRAVITINO_VERSION} ``` +On top of the [base GVFS configuration](./how-to-use-gvfs.md#configuration-1), pass the Azure Data Lake Storage +properties in `options`, spelled with underscores. + ```python from gravitino import gvfs + options = { "cache_size": 20, "cache_expired_time": 3600, "auth_type": "simple", - "azure_storage_account_name": "azure_account_name", - "azure_storage_account_key": "azure_account_key" + "azure_storage_account_name": "account_name", + "azure_storage_account_key": "account_key", } -fs = gvfs.GravitinoVirtualFileSystem(server_uri="http://localhost:8090", metalake_name="test_metalake", options=options) -fs.ls("gvfs://fileset/{adls_catalog}/{adls_schema}/{adls_fileset}/") -``` +fs = gvfs.GravitinoVirtualFileSystem(server_uri="http://localhost:8090", + metalake_name="metalake", + options=options) +fs.ls("gvfs://fileset/adls_catalog/adls_schema/example_fileset/") +``` -### Access the Fileset with Pandas +### pandas -The following are examples of how to use the pandas library to access the ADLS fileset +pandas reaches the same paths through `storage_options`. Use the `fs` instance from the preceding +GVFS example to discover the generated Spark part file. ```python import pandas as pd storage_options = { - "server_uri": "http://localhost:8090", - "metalake_name": "test", + "server_uri": "http://localhost:8090", + "metalake_name": "metalake", "options": { - "azure_storage_account_name": "azure_account_name", - "azure_storage_account_key": "azure_account_key" + "azure_storage_account_name": "account_name", + "azure_storage_account_key": "account_key", } } -ds = pd.read_csv(f"gvfs://fileset/${catalog_name}/${schema_name}/${fileset_name}/people/part-00000-51d366e2-d5eb-448d-9109-32a96c8a14dc-c000.csv", - storage_options=storage_options) + +csv_path = next( + f"gvfs://{path}" + for path in fs.ls( + "gvfs://fileset/adls_catalog/adls_schema/example_fileset/people", + detail=False, + ) + if ( + path.rsplit("/", 1)[-1].startswith("part-") + and path.endswith(".csv") + ) +) +ds = pd.read_csv(csv_path, storage_options=storage_options) ds.head() ``` -For other use cases, refer to the [Gravitino Virtual File System](./how-to-use-gvfs.md) document. +For further use cases, see [Gravitino Virtual File System](./how-to-use-gvfs.md). -## Fileset with Credential Vending +## Credential Vending -Gravitino supports credential vending for ADLS fileset. If the catalog has been [configured with credential](./security/credential-vending.md), you can access ADLS fileset without providing authentication information like `azure-storage-account-name` and `azure-storage-account-key` in the properties. +With credential vending the catalog holds the Azure Data Lake Storage credentials and the Gravitino server hands +out a credential per request, so clients never hold cloud keys of their own. See +[Credential Vending](./security/credential-vending.md) for the general mechanism and +[ADLS credentials](./security/credential-vending.md#adls) for the properties +each provider takes. -### Create an ADLS Fileset Catalog with Credential Vending +The supported providers are `adls-token`, which vends a short-lived token, and +`azure-account-key`, which vends the static account key configured on the catalog. The example below +uses `adls-token`. -Apart from configuration method in [create-adls-fileset-catalog](#adls-fileset-catalog-configuration), -properties needed by [adls-credential](./security/credential-vending.md#adls-credentials) should -also be set to enable credential vending for ADLS fileset. Take `adls-token` credential provider for example: +### Configure the catalog, schema, and fileset ```shell curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{ - "name": "adls-catalog-with-token", + "name": "adls_catalog_with_vending", "type": "FILESET", - "comment": "This is a ADLS fileset catalog", + "comment": "A fileset catalog backed by Azure Data Lake Storage with credential vending", "properties": { - "location": "abfss://container@account-name.dfs.core.windows.net/path", - "azure-storage-account-name": "The account name of the Azure Blob Storage", - "azure-storage-account-key": "The account key of the Azure Blob Storage", + "location": "abfss://container@account-name.dfs.core.windows.net/root", + "azure-storage-account-name": "account_name", + "azure-storage-account-key": "account_key", "credential-providers": "adls-token", - "azure-tenant-id":"The Azure tenant id", - "azure-client-id":"The Azure client id", - "azure-client-secret":"The Azure client secret key" + "azure-tenant-id": "The Azure tenant id", + "azure-client-id": "The Azure client id", + "azure-client-secret": "The Azure client secret key" } }' http://localhost:8090/api/metalakes/metalake/catalogs ``` -### Access an ADLS Fileset with Credential Vending +Create the schema and fileset in the credential-vending catalog: + +```shell +curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ +-H "Content-Type: application/json" -d '{ + "name": "adls_schema", + "comment": "A schema in the Azure Data Lake Storage credential-vending catalog", + "properties": { + "location": "abfss://container@account-name.dfs.core.windows.net/root/schema" + } +}' http://localhost:8090/api/metalakes/metalake/catalogs/adls_catalog_with_vending/schemas + +curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ +-H "Content-Type: application/json" -d '{ + "name": "example_fileset", + "comment": "This is an example fileset", + "type": "MANAGED", + "storageLocation": "abfss://container@account-name.dfs.core.windows.net/root/schema/example_fileset", + "properties": {} +}' http://localhost:8090/api/metalakes/metalake/catalogs/adls_catalog_with_vending/schemas/adls_schema/filesets +``` + +The `adls-token` provider needs three more catalog properties. + +| Property Name | Description | +|-----------------------|-------------------------| +| `azure-tenant-id` | Azure tenant id | +| `azure-client-id` | Azure client id | +| `azure-client-secret` | Azure client secret key | -When the catalog is configured with credentials and client-side credential vending is enabled, -you can access ADLS filesets directly using the GVFS Java/Python client or Spark without providing authentication details. +### Access without local credentials -GVFS Java client: +Enable vending on the client and drop the credential properties. ```java Configuration conf = new Configuration(); @@ -529,28 +544,34 @@ conf.setBoolean("fs.gravitino.enableCredentialVending", true); conf.set("fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs"); conf.set("fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem"); conf.set("fs.gravitino.server.uri", "http://localhost:8090"); -conf.set("fs.gravitino.client.metalake", "test_metalake"); -// No need to set azure-storage-account-name and azure-storage-account-name -Path filesetPath = new Path("gvfs://fileset/adls_test_catalog/test_schema/test_fileset/new_dir"); +conf.set("fs.gravitino.client.metalake", "metalake"); +// No need to set azure-storage-account-name or azure-storage-account-key + +Path filesetPath = new Path( + "gvfs://fileset/adls_catalog_with_vending/adls_schema/example_fileset/new_dir"); FileSystem fs = filesetPath.getFileSystem(conf); fs.mkdirs(filesetPath); -... ``` -Spark: - ```python -spark = SparkSession.builder - .appName("adls_fileset_test") +spark = (SparkSession.builder + .appName("adls_fileset") .config("spark.hadoop.fs.gravitino.enableCredentialVending", "true") .config("spark.hadoop.fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs") .config("spark.hadoop.fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem") .config("spark.hadoop.fs.gravitino.server.uri", "http://localhost:8090") - .config("spark.hadoop.fs.gravitino.client.metalake", "test") - # No need to set azure-storage-account-name and azure-storage-account-name - .config("spark.driver.memory", "2g") - .config("spark.driver.port", "2048") - .getOrCreate() + .config("spark.hadoop.fs.gravitino.client.metalake", "metalake") + # No need to set azure-storage-account-name or azure-storage-account-key + .getOrCreate()) ``` -Python client and Hadoop command are similar to the above examples. +```python +options = { + "auth_type": "simple", + "enable_credential_vending": True, + # No need to set azure-storage-account-name or azure-storage-account-key +} +fs = gvfs.GravitinoVirtualFileSystem(server_uri="http://localhost:8090", + metalake_name="metalake", + options=options) +``` diff --git a/docs/fileset-catalog-with-cos.md b/docs/fileset-catalog-with-cos.md index e20bd5f2032..dfe533d173c 100644 --- a/docs/fileset-catalog-with-cos.md +++ b/docs/fileset-catalog-with-cos.md @@ -1,68 +1,73 @@ --- title: "Fileset Catalog with COS" slug: "/fileset-catalog-with-cos" -date: 2026-6-17 -keyword: "Fileset catalog COS Tencent" +keyword: "Fileset catalog COS" license: "This software is licensed under the Apache License version 2." --- ## Introduction -This document explains how to configure a Fileset catalog with Tencent Cloud COS (Cloud Object Storage) in Gravitino. +This page shows how to store fileset data in Tencent Cloud COS while Gravitino manages the metadata, +and how to read and write that data through the Gravitino Virtual File System (GVFS). -## Prerequisites +Everything on this page is specific to Tencent Cloud COS. The fileset model itself, the properties shared by +every storage backend, and the way properties are inherited from catalog to schema to fileset are +described in [Fileset Catalog](./fileset-catalog.md). + +The examples run in order and use the same names throughout: metalake `metalake`, catalog +`cos_catalog`, schema `cos_schema`, fileset `example_fileset`, and `http://localhost:8090` as the +server URL. Replace them with your own values. -To set up a Fileset catalog with COS, follow these steps: +## Prerequisites 1. Download the [`gravitino-tencent-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-tencent-bundle) file. -2. Place the downloaded file into the Gravitino Fileset catalog classpath at `${GRAVITINO_HOME}/catalogs/fileset/libs/`. -3. Start the Gravitino server by running the following command: +2. Place it in the fileset catalog classpath at `${GRAVITINO_HOME}/catalogs/fileset/libs/`. +3. Start the Gravitino server: ```bash -$ ${GRAVITINO_HOME}/bin/gravitino-server.sh start +${GRAVITINO_HOME}/bin/gravitino-server.sh start ``` -Once the server is up and running, you can proceed to configure the Fileset catalog with COS. In the rest of this document we will use `http://localhost:8090` as the Gravitino server URL, replace with your actual server URL. +The catalog automatically loads the Tencent Cloud COS filesystem provider once the bundle jar is on the +classpath. The deprecated `filesystem-providers` and `default-filesystem-provider` catalog +properties do not need to be set. -## COS Catalog Configuration +## Tencent Cloud COS Properties -### COS Fileset Catalog Configuration +These properties are needed in addition to the shared +[catalog properties](./fileset-catalog.md#catalog-properties). The same values are also needed by +the GVFS clients, so they are listed together here — note that the Python client spells them with +underscores while the catalog and the Java client use hyphens. -In addition to the basic configurations mentioned in [Fileset catalog properties](./fileset-catalog.md#catalog-properties), the following properties are required to configure a Fileset catalog with COS: - -| Configuration item | Description | Default value | Required | -|-------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------| -| `cos-region` | The region of the Tencent Cloud COS bucket, e.g. `ap-guangzhou`, `ap-shanghai`. | (none) | Yes | -| `cos-endpoint` | The endpoint *suffix* of the Tencent Cloud COS service (mapped to `fs.cosn.bucket.endpoint_suffix`). It is a host suffix, not a full URL — e.g. `cos.ap-guangzhou.myqcloud.com`, **not** `https://cos.ap-guangzhou.myqcloud.com`. Optional; when not set, hadoop-cos derives it from `cos-region` (`cos.${region}.myqcloud.com`). Set this only if you need to point to a non-public endpoint (e.g. an internal/VPC endpoint). | (none) | No | -| `cos-access-key-id` | The static access key ID (Tencent Cloud `SecretId`) used to access COS data. | (none) | Yes | -| `cos-secret-access-key` | The static secret access key (Tencent Cloud `SecretKey`) used to access COS data. | (none) | Yes | -| `credential-providers` | The credential provider types, separated by comma. The currently supported value is `cos-secret-key`. Setting this enables credential vending provided by the Gravitino server, so the GVFS client no longer needs `cos-access-key-id` / `cos-secret-access-key` locally. See [cos-credential-vending](#fileset-with-credential-vending) below for details. | (none) | No | +| Catalog and Java client | Python client | Description | Required | +|-------------------------|-------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------| +| `cos-region` | `cos_region` | Region of the COS bucket, for example `ap-guangzhou` or `ap-shanghai`. | Yes | +| `cos-endpoint` | `cos_endpoint` | Endpoint *suffix* of the COS service, mapped to `fs.cosn.bucket.endpoint_suffix`. It is a host suffix, not a URL — `cos.ap-guangzhou.myqcloud.com`, not `https://cos.ap-guangzhou.myqcloud.com`. When unset, hadoop-cos derives it from `cos-region`. Set it only to reach a non-public endpoint such as a VPC endpoint. | No | +| `cos-access-key-id` | `cos_access_key_id` | Static access key id, the Tencent Cloud `SecretId`. | Yes | +| `cos-secret-access-key` | `cos_secret_access_key` | Static secret access key, the Tencent Cloud `SecretKey`. | Yes | +| `credential-providers` | (n/a) | The credential provider types, separated by comma. Possible values are `cos-secret-key`. Setting it enables credential vending, so clients no longer need the credentials above. See [credential vending](./security/credential-vending.md) for the extra properties each provider takes. | No | :::note -The fileset catalog automatically loads filesystem providers on the classpath. The COS provider -is registered when the `gravitino-tencent-bundle` jar is present, so `default-filesystem-provider` -and `filesystem-providers` do not need to be set. +`default-filesystem-provider` and `filesystem-providers` are deprecated. The fileset catalog +automatically loads the filesystem providers found on the classpath, including the built-in +providers and the cloud providers carried by a bundle jar such as `gravitino-tencent-bundle`. +::: +:::note `cos-region` is mandatory for hadoop-cos: signing requests, building the default endpoint and -selecting the right CAM scope all require the region. Even if you also set `cos-endpoint`, please -keep `cos-region` set. +selecting the right CAM scope all need the region. Keep it set even when `cos-endpoint` is also set. ::: -### Schema Configuration +Schema and fileset properties are documented on the shared page: see +[schema properties](./fileset-catalog.md#schema-properties) and +[fileset properties](./fileset-catalog.md#fileset-properties). -To create a schema, refer to [Schema configurations](./fileset-catalog.md#schema-properties). - -### Fileset Configuration - -For instructions on how to create a fileset, refer to [Fileset configurations](./fileset-catalog.md#fileset-properties) for more details. +A fileset catalog stores its data under `location`, which for Tencent Cloud COS looks like +`cosn://my-bucket-1250000000/root`. ## Create the Catalog, Schema, and Fileset -This section will show you how to use the Fileset catalog with COS in Gravitino, including detailed examples. - -### Step 1: Create a Fileset Catalog with COS - -First, you need to create a Fileset catalog for COS. The following examples demonstrate how to create a Fileset catalog with COS: +### Step 1: Create the catalog @@ -70,12 +75,13 @@ First, you need to create a Fileset catalog for COS. The following examples demo ```shell curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{ - "name": "test_catalog", + "name": "cos_catalog", "type": "FILESET", - "comment": "This is a COS fileset catalog", + "comment": "A fileset catalog backed by Tencent Cloud COS", "properties": { "location": "cosn://my-bucket-1250000000/root", "cos-region": "ap-guangzhou", + "cos-endpoint": "cos.ap-guangzhou.myqcloud.com", "cos-access-key-id": "access_key", "cos-secret-access-key": "secret_key" } @@ -91,50 +97,46 @@ GravitinoClient gravitinoClient = GravitinoClient .withMetalake("metalake") .build(); -Map cosProperties = ImmutableMap.builder() +Map catalogProperties = ImmutableMap.builder() .put("location", "cosn://my-bucket-1250000000/root") .put("cos-region", "ap-guangzhou") + .put("cos-endpoint", "cos.ap-guangzhou.myqcloud.com") .put("cos-access-key-id", "access_key") .put("cos-secret-access-key", "secret_key") .build(); -Catalog cosCatalog = gravitinoClient.createCatalog("test_catalog", - Type.FILESET, - "This is a COS fileset catalog", - cosProperties); -// ... - +Catalog catalog = gravitinoClient.createCatalog("cos_catalog", + Catalog.Type.FILESET, + "A fileset catalog backed by Tencent Cloud COS", + catalogProperties); ``` ```python -gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake") -cos_properties = { +gravitino_client: GravitinoClient = GravitinoClient( + uri="http://localhost:8090", metalake_name="metalake") + +catalog_properties = { "location": "cosn://my-bucket-1250000000/root", "cos-region": "ap-guangzhou", + "cos-endpoint": "cos.ap-guangzhou.myqcloud.com", "cos-access-key-id": "access_key", - "cos-secret-access-key": "secret_key" + "cos-secret-access-key": "secret_key", } -cos_catalog = gravitino_client.create_catalog(name="test_catalog", - catalog_type=Catalog.Type.FILESET, - provider=None, - comment="This is a COS fileset catalog", - properties=cos_properties) +catalog = gravitino_client.create_catalog(name="cos_catalog", + catalog_type=Catalog.Type.FILESET, + provider=None, + comment="A fileset catalog backed by Tencent Cloud COS", + properties=catalog_properties) ``` -:::note -Tencent Cloud COS bucket names always end with the appid suffix, for example `my-bucket-1250000000`. Use that full bucket name in `location` and in any `cosn://` URI. -::: - -### Step 2: Create a Schema - -Once the Fileset catalog with COS is created, you can create a schema inside that catalog. Below are examples of how to do this: +### Step 2: Create the schema @@ -142,49 +144,44 @@ Once the Fileset catalog with COS is created, you can create a schema inside tha ```shell curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{ - "name": "test_schema", - "comment": "This is a COS schema", + "name": "cos_schema", + "comment": "A schema in the Tencent Cloud COS fileset catalog", "properties": { "location": "cosn://my-bucket-1250000000/root/schema" } -}' http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas +}' http://localhost:8090/api/metalakes/metalake/catalogs/cos_catalog/schemas ``` ```java -Catalog catalog = gravitinoClient.loadCatalog("test_catalog"); - +Catalog catalog = gravitinoClient.loadCatalog("cos_catalog"); SupportsSchemas supportsSchemas = catalog.asSchemas(); Map schemaProperties = ImmutableMap.builder() .put("location", "cosn://my-bucket-1250000000/root/schema") .build(); -Schema schema = supportsSchemas.createSchema("test_schema", - "This is a COS schema", - schemaProperties -); -// ... + +Schema schema = supportsSchemas.createSchema("cos_schema", + "A schema in the Tencent Cloud COS fileset catalog", + schemaProperties); ``` ```python -gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake") -catalog: Catalog = gravitino_client.load_catalog(name="test_catalog") -catalog.as_schemas().create_schema(name="test_schema", - comment="This is a COS schema", +catalog: Catalog = gravitino_client.load_catalog(name="cos_catalog") +catalog.as_schemas().create_schema(name="cos_schema", + comment="A schema in the Tencent Cloud COS fileset catalog", properties={"location": "cosn://my-bucket-1250000000/root/schema"}) ``` -### Step 3: Create a Fileset - -Now that the schema is created, you can create a fileset inside it. Here's how: +### Step 3: Create the fileset @@ -199,303 +196,285 @@ curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ "properties": { "k1": "v1" } -}' http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas/test_schema/filesets +}' http://localhost:8090/api/metalakes/metalake/catalogs/cos_catalog/schemas/cos_schema/filesets ``` ```java -GravitinoClient gravitinoClient = GravitinoClient - .builder("http://localhost:8090") - .withMetalake("metalake") - .build(); - -Catalog catalog = gravitinoClient.loadCatalog("test_catalog"); +Catalog catalog = gravitinoClient.loadCatalog("cos_catalog"); FilesetCatalog filesetCatalog = catalog.asFilesetCatalog(); -Map propertiesMap = ImmutableMap.builder() - .put("k1", "v1") - .build(); +Map filesetProperties = ImmutableMap.builder() + .put("k1", "v1") + .build(); filesetCatalog.createFileset( - NameIdentifier.of("test_schema", "example_fileset"), + NameIdentifier.of("cos_schema", "example_fileset"), "This is an example fileset", Fileset.Type.MANAGED, "cosn://my-bucket-1250000000/root/schema/example_fileset", - propertiesMap); + filesetProperties); ``` ```python -gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake") - -catalog: Catalog = gravitino_client.load_catalog(name="test_catalog") -catalog.as_fileset_catalog().create_fileset(ident=NameIdentifier.of("test_schema", "example_fileset"), - type=Fileset.Type.MANAGED, - comment="This is an example fileset", - storage_location="cosn://my-bucket-1250000000/root/schema/example_fileset", - properties={"k1": "v1"}) +catalog: Catalog = gravitino_client.load_catalog(name="cos_catalog") +catalog.as_fileset_catalog().create_fileset( + ident=NameIdentifier.of("cos_schema", "example_fileset"), + type=Fileset.Type.MANAGED, + comment="This is an example fileset", + storage_location="cosn://my-bucket-1250000000/root/schema/example_fileset", + properties={"k1": "v1"}) ``` -## Access a Fileset with COS +The fileset is now addressable as +`gvfs://fileset/cos_catalog/cos_schema/example_fileset` from any GVFS client. -### Access the Fileset with the GVFS Java Client +## Access the Fileset -To access fileset with COS using the GVFS Java client, based on the [basic GVFS configurations](./how-to-use-gvfs.md#configuration-1), you need to add the following configurations: +### Java client jars -| Configuration item | Description | Default value | Required | -|-------------------------|--------------------------------------------------------------------------------------------------------------------------|---------------|----------| -| `cos-region` | The region of the Tencent Cloud COS bucket. | (none) | Yes | -| `cos-endpoint` | The endpoint *suffix* of the Tencent Cloud COS service (e.g. `cos.ap-guangzhou.myqcloud.com`, not a full URL). Optional. | (none) | No | -| `cos-access-key-id` | The access key ID (Tencent Cloud SecretId) for COS data. | (none) | Yes | -| `cos-secret-access-key` | The secret access key (Tencent Cloud SecretKey) for COS data. | (none) | Yes | +Every Java or Hadoop-based client needs `gravitino-filesystem-hadoop3-runtime`, which is published +on Maven Central, plus the Tencent Cloud COS filesystem implementation. Only the latter differs by +environment: + +| Environment | Jar providing the Tencent Cloud COS filesystem | +|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| No Hadoop installed | [`gravitino-tencent-bundle`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-tencent-bundle), a fat jar bundling `hadoop-cos` and the Tencent Cloud COS Java SDK | +| Hadoop already present | `hadoop-cos-3.3.0-8.3.23.jar` and `cos_api-bundle-5.6.227.jar`, published by Tencent Cloud on Maven Central and, unlike `hadoop-aws` or `hadoop-aliyun`, not part of the Apache Hadoop distribution | + +The artifacts in full: + +- [`gravitino-tencent-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-tencent-bundle): + a "fat" jar that includes the `gravitino-tencent` functionality together with every dependency it needs, + such as `hadoop-cos` and the Tencent Cloud COS Java SDK. Use it when the environment has no pre-existing Hadoop setup. +- [`gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-filesystem-hadoop3-runtime): + a "fat" jar that bundles the Gravitino virtual filesystem client and already includes the + `gravitino-tencent` functionality. Java and Hadoop-based clients require it to access Gravitino + filesets. +- `hadoop-cos-3.3.0-8.3.23.jar` and `cos_api-bundle-5.6.227.jar`: the standard Hadoop dependencies + for Tencent Cloud COS access, published by Tencent Cloud on Maven Central and, unlike `hadoop-aws` + or `hadoop-aliyun`, not part of the Apache Hadoop distribution. Supply them yourself when running + inside an existing Hadoop environment. + +```xml + + + org.apache.gravitino + gravitino-tencent-bundle + ${GRAVITINO_VERSION} + + + org.apache.gravitino + gravitino-filesystem-hadoop3-runtime + ${GRAVITINO_VERSION} + +``` + +```xml + + + org.apache.hadoop + hadoop-common + ${HADOOP_VERSION} + + + + com.qcloud.cos + hadoop-cos + 3.3.0-8.3.23 + + + org.apache.gravitino + gravitino-filesystem-hadoop3-runtime + ${GRAVITINO_VERSION} + +``` :::note -If the catalog has enabled [credential vending](security/credential-vending.md), the AK/SK properties above can be omitted. More details can be found in [Fileset with credential vending](#fileset-with-credential-vending). +The thin `gravitino-tencent` jar is not needed. Its functionality is already included in both +`gravitino-tencent-bundle` and `gravitino-filesystem-hadoop3-runtime`. ::: +### GVFS Java client + +On top of the [base GVFS configuration](./how-to-use-gvfs.md#configuration), set the Tencent Cloud COS +properties from the table above. + ```java Configuration conf = new Configuration(); conf.set("fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs"); conf.set("fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem"); conf.set("fs.gravitino.server.uri", "http://localhost:8090"); -conf.set("fs.gravitino.client.metalake", "test_metalake"); +conf.set("fs.gravitino.client.metalake", "metalake"); conf.set("cos-region", "ap-guangzhou"); +conf.set("cos-endpoint", "cos.ap-guangzhou.myqcloud.com"); conf.set("cos-access-key-id", "access_key"); conf.set("cos-secret-access-key", "secret_key"); -Path filesetPath = new Path("gvfs://fileset/test_catalog/test_schema/test_fileset/new_dir"); + +Path filesetPath = new Path("gvfs://fileset/cos_catalog/cos_schema/example_fileset/new_dir"); FileSystem fs = filesetPath.getFileSystem(conf); fs.mkdirs(filesetPath); -... -``` - -Similar to Spark configurations, you need to add COS (bundle) jars to the classpath according to your environment. -If you want to customise your hadoop version or there is already a hadoop version in your project, you can add the following dependencies to your `pom.xml`: - -```xml - - org.apache.hadoop - hadoop-common - ${HADOOP_VERSION} - - - - - com.qcloud.cos - hadoop-cos - 3.3.0-8.3.23 - - - - org.apache.gravitino - gravitino-filesystem-hadoop3-runtime - ${GRAVITINO_VERSION} - ``` -:::note -Unlike the S3, OSS, GCS and Azure connectors, COS does **not** ship with Apache Hadoop. The HCFS adapter for COS is published by Tencent Cloud as `com.qcloud.cos:hadoop-cos`. Make sure the version you pick is compatible with your Hadoop version (the `-` form encodes both, e.g. `3.3.0-8.3.23` targets Hadoop 3.3.0). -::: - -Or use the bundle jar with Hadoop environment if there is no Hadoop environment: +### Apache Spark -```xml - - org.apache.gravitino - gravitino-tencent-bundle - ${GRAVITINO_VERSION} - - - - org.apache.gravitino - gravitino-filesystem-hadoop3-runtime - ${GRAVITINO_VERSION} - -``` - -### Access the Fileset with Spark - -The following code snippet shows how to use **PySpark 3.5.0 with Hadoop environment(Hadoop 3.3.4)** to access the fileset: - -Before running the following code, you need to install required packages: +The example below uses PySpark 3.5.0 in an environment that already has Hadoop 3.3.4. ```bash pip install pyspark==3.5.0 pip install apache-gravitino==${GRAVITINO_VERSION} ``` -Then you can run the following code: ```python -from pyspark.sql import SparkSession import os +from pyspark.sql import SparkSession -gravitino_url = "http://localhost:8090" -metalake_name = "test" - -catalog_name = "your_cos_catalog" -schema_name = "your_cos_schema" -fileset_name = "your_cos_fileset" - -# JDK8 as follows. JDK17 will be slightly different, you need to add -# '--conf "spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" -# --conf "spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"' -# to the submit args. +# On JDK 17, also add: +# --conf "spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" +# --conf "spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" os.environ["PYSPARK_SUBMIT_ARGS"] = ( - "--jars " - "/path/to/gravitino-filesystem-hadoop3-runtime-{gravitino-version}.jar," + "--jars /path/to/gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar," "/path/to/hadoop-cos-3.3.0-8.3.23.jar," "/path/to/cos_api-bundle-5.6.227.jar " "--master local[1] pyspark-shell" ) -spark = SparkSession.builder \ - .appName("cos_fileset_test") \ - .config("spark.hadoop.fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs") \ - .config("spark.hadoop.fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem") \ - .config("spark.hadoop.fs.gravitino.server.uri", gravitino_url) \ - .config("spark.hadoop.fs.gravitino.client.metalake", "test") \ - .config("spark.hadoop.cos-region", "ap-guangzhou") \ - .config("spark.hadoop.cos-access-key-id", os.environ["COS_ACCESS_KEY_ID"]) \ - .config("spark.hadoop.cos-secret-access-key", os.environ["COS_SECRET_ACCESS_KEY"]) \ - .config("spark.driver.memory", "2g") \ - .config("spark.driver.port", "2048") \ - .getOrCreate() + +spark = (SparkSession.builder + .appName("cos_fileset") + .config("spark.hadoop.fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs") + .config("spark.hadoop.fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem") + .config("spark.hadoop.fs.gravitino.server.uri", "http://localhost:8090") + .config("spark.hadoop.fs.gravitino.client.metalake", "metalake") + .config("spark.hadoop.cos-region", "ap-guangzhou") + .config("spark.hadoop.cos-endpoint", "cos.ap-guangzhou.myqcloud.com") + .config("spark.hadoop.cos-access-key-id", "access_key") + .config("spark.hadoop.cos-secret-access-key", "secret_key") + .config("spark.driver.memory", "2g") + .config("spark.driver.port", "2048") + .getOrCreate()) data = [("Alice", 25), ("Bob", 30), ("Cathy", 45)] -columns = ["Name", "Age"] -spark_df = spark.createDataFrame(data, schema=columns) -gvfs_path = f"gvfs://fileset/{catalog_name}/{schema_name}/{fileset_name}/people" - -spark_df.coalesce(1).write \ - .mode("overwrite") \ - .option("header", "true") \ - .csv(gvfs_path) +spark_df = spark.createDataFrame(data, schema=["Name", "Age"]) +gvfs_path = "gvfs://fileset/cos_catalog/cos_schema/example_fileset/people" + +spark_df.coalesce(1).write.mode("overwrite").option("header", "true").csv(gvfs_path) ``` -If your Spark is **without Hadoop environment**, you can use the following code snippet to access the fileset: +If Spark runs without a Hadoop environment, only the jar list changes: ```python -## Only the PYSPARK_SUBMIT_ARGS line below changes; keep all the SparkSession.builder.config(...) -## calls (including spark.hadoop.cos-region / cos-access-key-id / cos-secret-access-key) above as-is. - -os.environ["PYSPARK_SUBMIT_ARGS"] = "--jars /path/to/gravitino-tencent-bundle-{gravitino-version}.jar,/path/to/gravitino-filesystem-hadoop3-runtime-{gravitino-version}.jar --master local[1] pyspark-shell" +os.environ["PYSPARK_SUBMIT_ARGS"] = ( + "--jars /path/to/gravitino-tencent-bundle-${gravitino-version}.jar," + "/path/to/gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar " + "--master local[1] pyspark-shell" +) ``` -- [`gravitino-tencent-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-tencent-bundle): A "fat" JAR that includes `gravitino-tencent` functionality and all necessary dependencies like `hadoop-cos` and the Tencent Cloud COS Java SDK. Use this if your Spark environment doesn't have a pre-existing Hadoop setup. -- [`gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-filesystem-hadoop3-runtime): A "fat" JAR that bundles Gravitino's virtual filesystem client and includes the functionality of `gravitino-tencent`. It is required for accessing Gravitino filesets. - -Please choose the correct jar according to your environment. - :::note -In some Spark versions, a Hadoop environment is needed by the driver, adding the bundle jars with `--jars` may not work. If this is the case, you should add the jars to the spark CLASSPATH directly. +Some Spark versions need a Hadoop environment in the driver and do not pick up filesystem +implementations passed with `--jars`. If that happens, add the jars to the Spark classpath directly. ::: -### Access a Fileset Using the Hadoop Fs Command +### Hadoop fs command -The following are examples of how to use the `hadoop fs` command to access the fileset in Hadoop 3.1.3: - -1. Add the following contents to the `${HADOOP_HOME}/etc/hadoop/core-site.xml` file: +1. Add the following to `${HADOOP_HOME}/etc/hadoop/core-site.xml`: ```xml - - fs.AbstractFileSystem.gvfs.impl - org.apache.gravitino.filesystem.hadoop.Gvfs - - - - fs.gvfs.impl - org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem - - - - fs.gravitino.server.uri - http://localhost:8090 - - - - fs.gravitino.client.metalake - test - - - - cos-region - ap-guangzhou - - - - cos-access-key-id - access-key - - - - cos-secret-access-key - secret-key - + + fs.AbstractFileSystem.gvfs.impl + org.apache.gravitino.filesystem.hadoop.Gvfs + + + fs.gvfs.impl + org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem + + + fs.gravitino.server.uri + http://localhost:8090 + + + fs.gravitino.client.metalake + metalake + + + cos-region + ap-guangzhou + + + cos-endpoint + cos.ap-guangzhou.myqcloud.com + + + cos-access-key-id + access_key + + + cos-secret-access-key + secret_key + ``` -2. Add the necessary jars to the Hadoop classpath. +2. Add these jars to the Hadoop classpath: -For COS, you need to add `gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`, `hadoop-cos-3.3.0-8.3.23.jar` and `cos_api-bundle-5.6.227.jar` to the Hadoop classpath. Unlike `hadoop-aws` or `hadoop-aliyun`, these jars are *not* part of the Apache Hadoop distribution; download them from Maven Central and place them under `${HADOOP_HOME}/share/hadoop/tools/lib/`. + - `gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`, from Maven Central. + - `hadoop-cos-3.3.0-8.3.23.jar` and `cos_api-bundle-5.6.227.jar`, published by Tencent Cloud on Maven Central and, unlike `hadoop-aws` or `hadoop-aliyun`, not part of the Apache Hadoop distribution. -3. Run the following command to access the fileset: +3. Access the fileset: ```shell -./${HADOOP_HOME}/bin/hadoop fs -ls gvfs://fileset/cos_catalog/cos_schema/cos_fileset -./${HADOOP_HOME}/bin/hadoop fs -put /path/to/local/file gvfs://fileset/cos_catalog/cos_schema/cos_fileset +${HADOOP_HOME}/bin/hadoop fs -ls gvfs://fileset/cos_catalog/cos_schema/example_fileset +${HADOOP_HOME}/bin/hadoop fs -put /path/to/local/file gvfs://fileset/cos_catalog/cos_schema/example_fileset ``` -### Access the Fileset with the GVFS Python Client / Pandas +### GVFS Python client and pandas :::note -The GVFS **Python** client does not yet ship a COS storage handler, so reading -and writing COS-backed filesets via the Python `gvfs.GravitinoVirtualFileSystem` -or pandas `read_csv("gvfs://...")` is **not supported in this release**. Only -the GVFS **Java** client (and Spark / `hadoop fs` on top of it) can do fileset -I/O against COS today. - -The Python `GravitinoClient` itself is not affected: you can still create, -inspect, update and delete COS catalogs, schemas and filesets through the -metadata API exactly as shown in the [Step 1 example above](#step-1-create-a-fileset-catalog-with-cos) and the following Step 2 / Step 3 sections. -What is missing is purely the *data plane* (`gvfs.ls`, `gvfs.open`, -pandas `read_csv("gvfs://...")`, etc.) for COS, which will be added in a -follow-up release together with a `COSStorageHandler` and the corresponding -fsspec adapter. -::: +The GVFS Python client does not yet ship a COS storage handler. It cannot read or write +COS-backed filesets through `gvfs.GravitinoVirtualFileSystem` or pandas +`read_csv("gvfs://...")`. Use the GVFS Java client, Spark, or `hadoop fs` for COS data access. -For other use cases of the Java GVFS client, refer to the [Gravitino Virtual File System](./how-to-use-gvfs.md) document. +This limitation does not affect the Python `GravitinoClient` metadata API. It can still create, +inspect, update, and delete COS catalogs, schemas, and filesets as shown in +[Create the Catalog, Schema, and Fileset](#create-the-catalog-schema-and-fileset). +::: -## Fileset with Credential Vending +For further Java client use cases, see +[Gravitino Virtual File System](./how-to-use-gvfs.md). -Gravitino supports credential vending for the COS fileset. If the catalog has been [configured with credential](./security/credential-vending.md), you can access COS fileset without providing authentication information like `cos-access-key-id` and `cos-secret-access-key` in the GVFS client configuration. +## Credential Vending -The currently supported credential providers are listed below: +With credential vending the catalog holds the Tencent Cloud COS credentials and the Gravitino server hands +out a credential per request, so clients never hold cloud keys of their own. See +[Credential Vending](./security/credential-vending.md) for the general mechanism. -| Credential provider | Description | Vended credential type | -|---------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------| -| `cos-secret-key` | The Gravitino server hands out the static `cos-access-key-id` / `cos-secret-access-key` configured on the catalog. Useful for centralising credentials on the server side. | Static AK/SK | +The supported provider is `cos-secret-key`, which vends the static +`cos-access-key-id` / `cos-secret-access-key` configured on the catalog. :::note -STS-token-based credential vending (`cos-token`) for COS, including role-arn / assume-role configuration, will be added in a follow-up release. Until then, only the static `cos-secret-key` provider is available. +STS-token-based vending (`cos-token`), including role-arn and assume-role configuration, will be +added in a follow-up release. Until then only `cos-secret-key` is available. ::: -### Create a COS Fileset Catalog with Credential Vending - -In addition to the configuration described in [COS fileset catalog configuration](#cos-fileset-catalog-configuration), you only need to set `credential-providers` to `cos-secret-key` to enable credential vending: +### Configure the catalog, schema, and fileset ```shell curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{ - "name": "cos-catalog-with-credential-vending", + "name": "cos_catalog_with_vending", "type": "FILESET", - "comment": "This is a COS fileset catalog with credential vending", + "comment": "A fileset catalog backed by Tencent Cloud COS with credential vending", "properties": { "location": "cosn://my-bucket-1250000000/root", "cos-region": "ap-guangzhou", + "cos-endpoint": "cos.ap-guangzhou.myqcloud.com", "cos-access-key-id": "access_key", "cos-secret-access-key": "secret_key", "credential-providers": "cos-secret-key" @@ -503,12 +482,31 @@ curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ }' http://localhost:8090/api/metalakes/metalake/catalogs ``` -### Access a COS Fileset with Credential Vending +Create the schema and fileset in the credential-vending catalog: -When the catalog is configured with credentials and client-side credential vending is enabled, -you can access COS filesets directly using the GVFS Java client or Spark without providing AK/SK in the client. +```shell +curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ +-H "Content-Type: application/json" -d '{ + "name": "cos_schema", + "comment": "A schema in the Tencent Cloud COS credential-vending catalog", + "properties": { + "location": "cosn://my-bucket-1250000000/root/schema" + } +}' http://localhost:8090/api/metalakes/metalake/catalogs/cos_catalog_with_vending/schemas -GVFS Java client: +curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ +-H "Content-Type: application/json" -d '{ + "name": "example_fileset", + "comment": "This is an example fileset", + "type": "MANAGED", + "storageLocation": "cosn://my-bucket-1250000000/root/schema/example_fileset", + "properties": {} +}' http://localhost:8090/api/metalakes/metalake/catalogs/cos_catalog_with_vending/schemas/cos_schema/filesets +``` + +### Access without local credentials + +Enable vending on the client and drop the credential properties. ```java Configuration conf = new Configuration(); @@ -516,29 +514,26 @@ conf.setBoolean("fs.gravitino.enableCredentialVending", true); conf.set("fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs"); conf.set("fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem"); conf.set("fs.gravitino.server.uri", "http://localhost:8090"); -conf.set("fs.gravitino.client.metalake", "test_metalake"); -// No need to set cos-access-key-id and cos-secret-access-key -Path filesetPath = new Path("gvfs://fileset/cos_test_catalog/test_schema/test_fileset/new_dir"); +conf.set("fs.gravitino.client.metalake", "metalake"); +// No need to set cos-access-key-id or cos-secret-access-key + +Path filesetPath = new Path( + "gvfs://fileset/cos_catalog_with_vending/cos_schema/example_fileset/new_dir"); FileSystem fs = filesetPath.getFileSystem(conf); fs.mkdirs(filesetPath); -... ``` -Spark: - ```python -spark = SparkSession.builder \ - .appName("cos_fileset_test") \ - .config("spark.hadoop.fs.gravitino.enableCredentialVending", "true") \ - .config("spark.hadoop.fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs") \ - .config("spark.hadoop.fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem") \ - .config("spark.hadoop.fs.gravitino.server.uri", "http://localhost:8090") \ - .config("spark.hadoop.fs.gravitino.client.metalake", "test") \ - .config("spark.driver.memory", "2g") \ - .config("spark.driver.port", "2048") \ - .getOrCreate() -# Note: no need to set spark.hadoop.cos-access-key-id / cos-secret-access-key here — -# credential vending will fetch them from the Gravitino server. +spark = (SparkSession.builder + .appName("cos_fileset") + .config("spark.hadoop.fs.gravitino.enableCredentialVending", "true") + .config("spark.hadoop.fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs") + .config("spark.hadoop.fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem") + .config("spark.hadoop.fs.gravitino.server.uri", "http://localhost:8090") + .config("spark.hadoop.fs.gravitino.client.metalake", "metalake") + # No need to set cos-access-key-id or cos-secret-access-key + .getOrCreate()) ``` -The Hadoop `fs` command is similar to the above examples. The GVFS **Python** client cannot access COS-backed filesets yet — see the note in [Access the Fileset with the GVFS Python Client / Pandas](#access-the-fileset-with-the-gvfs-python-client--pandas). +The GVFS Python client cannot access COS-backed filesets; see +[GVFS Python client and pandas](#gvfs-python-client-and-pandas). diff --git a/docs/fileset-catalog-with-gcs.md b/docs/fileset-catalog-with-gcs.md index 8e2dae9d99a..68a8b41eb4a 100644 --- a/docs/fileset-catalog-with-gcs.md +++ b/docs/fileset-catalog-with-gcs.md @@ -1,61 +1,64 @@ --- title: "Fileset Catalog with GCS" slug: "/fileset-catalog-with-gcs" -date: 2024-01-03 keyword: "Fileset catalog GCS" license: "This software is licensed under the Apache License version 2." --- ## Introduction -This document describes how to configure a Fileset catalog with GCS. +This page shows how to store fileset data in Google Cloud Storage while Gravitino manages the metadata, +and how to read and write that data through the Gravitino Virtual File System (GVFS). -## Prerequisites +Everything on this page is specific to Google Cloud Storage. The fileset model itself, the properties shared by +every storage backend, and the way properties are inherited from catalog to schema to fileset are +described in [Fileset Catalog](./fileset-catalog.md). + +The examples run in order and use the same names throughout: metalake `metalake`, catalog +`gcs_catalog`, schema `gcs_schema`, fileset `example_fileset`, and `http://localhost:8090` as the +server URL. Replace them with your own values. -To set up a Fileset catalog with OSS, follow these steps: +## Prerequisites 1. Download the [`gravitino-gcp-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-gcp-bundle) file. -2. Place the downloaded file into the Gravitino Fileset catalog classpath at `${GRAVITINO_HOME}/catalogs/fileset/libs/`. -3. Start the Gravitino server by running the following command: +2. Place it in the fileset catalog classpath at `${GRAVITINO_HOME}/catalogs/fileset/libs/`. +3. Start the Gravitino server: ```bash -$ ${GRAVITINO_HOME}/bin/gravitino-server.sh start +${GRAVITINO_HOME}/bin/gravitino-server.sh start ``` -Once the server is up and running, you can proceed to configure the Fileset catalog with GCS. In the rest of this document we will use `http://localhost:8090` as the Gravitino server URL, replace with your actual server URL. +The catalog automatically loads the Google Cloud Storage filesystem provider once the bundle jar is on the +classpath. The deprecated `filesystem-providers` and `default-filesystem-provider` catalog +properties do not need to be set. -## GCS Catalog Configuration +## Google Cloud Storage Properties -### GCS Fileset Catalog Configuration +These properties are needed in addition to the shared +[catalog properties](./fileset-catalog.md#catalog-properties). The same values are also needed by +the GVFS clients, so they are listed together here — note that the Python client spells them with +underscores while the catalog and the Java client use hyphens. -Apart from configurations mentioned in [Fileset-catalog-catalog-configuration](./fileset-catalog.md#catalog-properties), the following properties are required to configure a Fileset catalog with GCS: - -| Configuration item | Description | Default value | Required | -|-------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|----------| -| `filesystem-providers` | (deprecated) The file system providers to add. Set it to `gcs` if it's a GCS fileset, a comma separated string that contains `gcs` like `gcs,s3` to support multiple kinds of fileset including `gcs`. | (none) | Yes | -| `default-filesystem-provider` | (deprecated) The name default filesystem providers of this Fileset catalog if users do not specify the scheme in the URI. Default value is `builtin-local`, for GCS, if we set this value, we can omit the prefix 'gs://' in the location. | `builtin-local` | No | -| `gcs-service-account-file` | The path of GCS service account JSON file. | (none) | Yes | -| `credential-providers` | The credential provider types, separated by comma, possible value can be `gcs-token`. As the default authentication type is using service account as the above, this configuration can enable credential vending provided by Gravitino server and client will no longer need to provide authentication information like service account to access GCS by GVFS. Once it's set, more configuration items are needed to make it works, see [gcs-credential-vending](security/credential-vending.md#gcs-credentials) | (none) | No | +| Catalog and Java client | Python client | Description | Required | +|----------------------------|----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------| +| `gcs-service-account-file` | `gcs_service_account_file` | Path of the GCS service account JSON file. | Yes | +| `credential-providers` | (n/a) | The credential provider types, separated by comma. Possible values are `gcs-token`. Setting it enables credential vending, so clients no longer need the credentials above. See [credential vending](./security/credential-vending.md#gcs) for the extra properties each provider takes. | No | :::note -`default-filesystem-provider` and `filesystem-providers` are deprecated. The fileset catalog automatically loads filesystem providers on the classpath, including buildin filesystem provider and cloud providers when the corresponding bundle jar is present (for example, `gravitino-gcp-bundle`). +The service account file must be readable by the Gravitino server process for the catalog, and by +each client process for GVFS. ::: -### Schema Configuration - -Refer to [Schema configurations](./fileset-catalog.md#schema-properties) for more details. - -### Fileset Configuration +Schema and fileset properties are documented on the shared page: see +[schema properties](./fileset-catalog.md#schema-properties) and +[fileset properties](./fileset-catalog.md#fileset-properties). -Refer to [Fileset configurations](./fileset-catalog.md#fileset-properties) for more details. +A fileset catalog stores its data under `location`, which for Google Cloud Storage looks like +`gs://bucket/root`. ## Create the Catalog, Schema, and Fileset -This section will show you how to use the Fileset catalog with GCS in Gravitino, including detailed examples. - -### Step 1: Create a Fileset Catalog with GCS - -First, you need to create a Fileset catalog with GCS. The following example shows how to create a Fileset catalog with GCS: +### Step 1: Create the catalog @@ -63,12 +66,12 @@ First, you need to create a Fileset catalog with GCS. The following example show ```shell curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{ - "name": "test_catalog", + "name": "gcs_catalog", "type": "FILESET", - "comment": "This is a GCS fileset catalog", + "comment": "A fileset catalog backed by Google Cloud Storage", "properties": { "location": "gs://bucket/root", - "gcs-service-account-file": "path_of_gcs_service_account_file" + "gcs-service-account-file": "/path/to/service-account.json" } }' http://localhost:8090/api/metalakes/metalake/catalogs ``` @@ -82,42 +85,40 @@ GravitinoClient gravitinoClient = GravitinoClient .withMetalake("metalake") .build(); -Map gcsProperties = ImmutableMap.builder() +Map catalogProperties = ImmutableMap.builder() .put("location", "gs://bucket/root") - .put("gcs-service-account-file", "path_of_gcs_service_account_file") + .put("gcs-service-account-file", "/path/to/service-account.json") .build(); -Catalog gcsCatalog = gravitinoClient.createCatalog("test_catalog", - Type.FILESET, - "This is a GCS fileset catalog", - gcsProperties); -// ... - +Catalog catalog = gravitinoClient.createCatalog("gcs_catalog", + Catalog.Type.FILESET, + "A fileset catalog backed by Google Cloud Storage", + catalogProperties); ``` ```python -gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake") -gcs_properties = { +gravitino_client: GravitinoClient = GravitinoClient( + uri="http://localhost:8090", metalake_name="metalake") + +catalog_properties = { "location": "gs://bucket/root", - "gcs-service-account-file": "path_of_gcs_service_account_file" + "gcs-service-account-file": "/path/to/service-account.json", } -gcs_properties = gravitino_client.create_catalog(name="test_catalog", - catalog_type=Catalog.Type.FILESET, - provider=None, - comment="This is a GCS fileset catalog", - properties=gcs_properties) +catalog = gravitino_client.create_catalog(name="gcs_catalog", + catalog_type=Catalog.Type.FILESET, + provider=None, + comment="A fileset catalog backed by Google Cloud Storage", + properties=catalog_properties) ``` -### Step 2: Create a Schema - -Once you’ve created a Fileset catalog with GCS, you can create a schema. The following example shows how to create a schema: +### Step 2: Create the schema @@ -125,50 +126,44 @@ Once you’ve created a Fileset catalog with GCS, you can create a schema. The f ```shell curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{ - "name": "test_schema", - "comment": "This is a GCS schema", + "name": "gcs_schema", + "comment": "A schema in the Google Cloud Storage fileset catalog", "properties": { "location": "gs://bucket/root/schema" } -}' http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas +}' http://localhost:8090/api/metalakes/metalake/catalogs/gcs_catalog/schemas ``` ```java -Catalog catalog = gravitinoClient.loadCatalog("test_catalog"); - +Catalog catalog = gravitinoClient.loadCatalog("gcs_catalog"); SupportsSchemas supportsSchemas = catalog.asSchemas(); Map schemaProperties = ImmutableMap.builder() .put("location", "gs://bucket/root/schema") .build(); -Schema schema = supportsSchemas.createSchema("test_schema", - "This is a GCS schema", - schemaProperties -); -// ... + +Schema schema = supportsSchemas.createSchema("gcs_schema", + "A schema in the Google Cloud Storage fileset catalog", + schemaProperties); ``` ```python -gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake") -catalog: Catalog = gravitino_client.load_catalog(name="test_catalog") -catalog.as_schemas().create_schema(name="test_schema", - comment="This is a GCS schema", +catalog: Catalog = gravitino_client.load_catalog(name="gcs_catalog") +catalog.as_schemas().create_schema(name="gcs_schema", + comment="A schema in the Google Cloud Storage fileset catalog", properties={"location": "gs://bucket/root/schema"}) ``` - -### Step 3: Create a Fileset - -After creating a schema, you can create a fileset. The following example shows how to create a fileset: +### Step 3: Create the fileset @@ -183,325 +178,337 @@ curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ "properties": { "k1": "v1" } -}' http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas/test_schema/filesets +}' http://localhost:8090/api/metalakes/metalake/catalogs/gcs_catalog/schemas/gcs_schema/filesets ``` ```java -GravitinoClient gravitinoClient = GravitinoClient - .builder("http://localhost:8090") - .withMetalake("metalake") - .build(); - -Catalog catalog = gravitinoClient.loadCatalog("test_catalog"); +Catalog catalog = gravitinoClient.loadCatalog("gcs_catalog"); FilesetCatalog filesetCatalog = catalog.asFilesetCatalog(); -Map propertiesMap = ImmutableMap.builder() - .put("k1", "v1") - .build(); +Map filesetProperties = ImmutableMap.builder() + .put("k1", "v1") + .build(); filesetCatalog.createFileset( - NameIdentifier.of("test_schema", "example_fileset"), + NameIdentifier.of("gcs_schema", "example_fileset"), "This is an example fileset", Fileset.Type.MANAGED, "gs://bucket/root/schema/example_fileset", - propertiesMap, -); + filesetProperties); ``` ```python -gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake") - -catalog: Catalog = gravitino_client.load_catalog(name="test_catalog") -catalog.as_fileset_catalog().create_fileset(ident=NameIdentifier.of("test_schema", "example_fileset"), - type=Fileset.Type.MANAGED, - comment="This is an example fileset", - storage_location="gs://bucket/root/schema/example_fileset", - properties={"k1": "v1"}) +catalog: Catalog = gravitino_client.load_catalog(name="gcs_catalog") +catalog.as_fileset_catalog().create_fileset( + ident=NameIdentifier.of("gcs_schema", "example_fileset"), + type=Fileset.Type.MANAGED, + comment="This is an example fileset", + storage_location="gs://bucket/root/schema/example_fileset", + properties={"k1": "v1"}) ``` -## Access a Fileset with GCS +The fileset is now addressable as +`gvfs://fileset/gcs_catalog/gcs_schema/example_fileset` from any GVFS client. -### Access the Fileset with the GVFS Java Client +## Access the Fileset -To access fileset with GCS using the GVFS Java client, based on the [basic GVFS configurations](./how-to-use-gvfs.md#configuration-1), you need to add the following configurations: +### Java client jars -| Configuration item | Description | Default value | Required | -|----------------------------|--------------------------------------------|---------------|----------| -| `gcs-service-account-file` | The path of GCS service account JSON file. | (none) | Yes | +Every Java or Hadoop-based client needs `gravitino-filesystem-hadoop3-runtime`, which is published +on Maven Central, plus the Google Cloud Storage filesystem implementation. Only the latter differs +by environment: -:::note -If the catalog has enabled [credential vending](security/credential-vending.md), the properties above can be omitted. More details can be found in [Fileset with credential vending](#fileset-with-credential-vending). -::: +| Environment | Jar providing the Google Cloud Storage filesystem | +|------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| No Hadoop installed | [`gravitino-gcp-bundle`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-gcp-bundle), a fat jar bundling the Google Cloud Storage filesystem implementation and its dependencies | +| Hadoop already present | [`gcs-connector-hadoop3-2.2.22-shaded.jar`](https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/download/v2.2.22/gcs-connector-hadoop3-2.2.22-shaded.jar), published by Google and not part of the Apache Hadoop distribution | -```java -Configuration conf = new Configuration(); -conf.set("fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs"); -conf.set("fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem"); -conf.set("fs.gravitino.server.uri", "http://localhost:8090"); -conf.set("fs.gravitino.client.metalake", "test_metalake"); -conf.set("gcs-service-account-file", "/path/your-service-account-file.json"); -Path filesetPath = new Path("gvfs://fileset/test_catalog/test_schema/test_fileset/new_dir"); -FileSystem fs = filesetPath.getFileSystem(conf); -fs.mkdirs(filesetPath); -... -``` +The artifacts in full: + +- [`gravitino-gcp-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-gcp-bundle): + a "fat" jar that includes the `gravitino-gcp` functionality together with every dependency it needs, + such as `gcs-connector`. Use it when the environment has no pre-existing Hadoop setup. +- [`gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-filesystem-hadoop3-runtime): + a "fat" jar that bundles the Gravitino virtual filesystem client and already includes the + `gravitino-gcp` functionality. Java and Hadoop-based clients require it to access Gravitino + filesets. +- [`gcs-connector-hadoop3-2.2.22-shaded.jar`](https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/download/v2.2.22/gcs-connector-hadoop3-2.2.22-shaded.jar): + the standard Hadoop dependencies for Google Cloud Storage access, published by Google and not part + of the Apache Hadoop distribution. Supply them yourself when running inside an existing Hadoop + environment. +- [`gravitino-gcp-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-gcp): + a "thin" jar carrying only the GCP integration code. It is already contained in both jars above, + so it is not needed as a direct dependency unless you prefer to manage all Hadoop and GCP + dependencies yourself. -Similar to Spark configurations, you need to add GCS (bundle) jars to the classpath according to your environment. -If your wants to custom your hadoop version or there is already a hadoop version in your project, you can add the following dependencies to your `pom.xml`: +```xml + + + org.apache.gravitino + gravitino-gcp-bundle + ${GRAVITINO_VERSION} + + + org.apache.gravitino + gravitino-filesystem-hadoop3-runtime + ${GRAVITINO_VERSION} + +``` ```xml - - org.apache.hadoop - hadoop-common - ${HADOOP_VERSION} - - - com.google.cloud.bigdataoss - gcs-connector - ${GCS_CONNECTOR_VERSION} - - - org.apache.gravitino - gravitino-filesystem-hadoop3-runtime - ${GRAVITINO_VERSION} - + + + org.apache.hadoop + hadoop-common + ${HADOOP_VERSION} + + + com.google.cloud.bigdataoss + gcs-connector + hadoop3-2.2.22 + + + org.apache.gravitino + gravitino-filesystem-hadoop3-runtime + ${GRAVITINO_VERSION} + ``` :::note -The `gravitino-gcp` JAR is no longer required, as it is now included in the `gravitino-filesystem-hadoop3-runtime` JAR. +The thin `gravitino-gcp` jar is not needed. Its functionality is already included in both +`gravitino-gcp-bundle` and `gravitino-filesystem-hadoop3-runtime`. ::: -Or use the bundle jar with Hadoop environment if there is no Hadoop environment: +### GVFS Java client -```xml - - org.apache.gravitino - gravitino-gcp-bundle - ${GRAVITINO_VERSION} - - - - org.apache.gravitino - gravitino-filesystem-hadoop3-runtime - ${GRAVITINO_VERSION} - -``` +On top of the [base GVFS configuration](./how-to-use-gvfs.md#configuration), set the Google Cloud Storage +properties from the table above. -### Access the Fileset with Spark +```java +Configuration conf = new Configuration(); +conf.set("fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs"); +conf.set("fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem"); +conf.set("fs.gravitino.server.uri", "http://localhost:8090"); +conf.set("fs.gravitino.client.metalake", "metalake"); +conf.set("gcs-service-account-file", "/path/to/service-account.json"); + +Path filesetPath = new Path("gvfs://fileset/gcs_catalog/gcs_schema/example_fileset/new_dir"); +FileSystem fs = filesetPath.getFileSystem(conf); +fs.mkdirs(filesetPath); +``` -The following code snippet shows how to use **PySpark 3.5.0 with Hadoop environment(Hadoop 3.3.4)** to access the fileset: +### Apache Spark -Before running the following code, you need to install required packages: +The example below uses PySpark 3.5.0 in an environment that already has Hadoop 3.3.4. ```bash pip install pyspark==3.5.0 pip install apache-gravitino==${GRAVITINO_VERSION} ``` -Then you can run the following code: ```python -from pyspark.sql import SparkSession import os +from pyspark.sql import SparkSession -gravitino_url = "http://localhost:8090" -metalake_name = "test" - -catalog_name = "your_gcs_catalog" -schema_name = "your_gcs_schema" -fileset_name = "your_gcs_fileset" +# On JDK 17, also add: +# --conf "spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" +# --conf "spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" +os.environ["PYSPARK_SUBMIT_ARGS"] = ( + "--jars /path/to/gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar," + "/path/to/gcs-connector-hadoop3-2.2.22-shaded.jar " + "--master local[1] pyspark-shell" +) -# JDK8 as follows, JDK17 will be slightly different, you need to add '--conf \"spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED\" --conf \"spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED\"' to the submit args. -os.environ["PYSPARK_SUBMIT_ARGS"] = "--jars /path/to/gravitino-filesystem-hadoop3-runtime-{gravitino-version}.jar,/path/to/gcs-connector-hadoop3-2.2.22-shaded.jar --master local[1] pyspark-shell" -spark = SparkSession.builder - .appName("gcs_fielset_test") +spark = (SparkSession.builder + .appName("gcs_fileset") .config("spark.hadoop.fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs") .config("spark.hadoop.fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem") .config("spark.hadoop.fs.gravitino.server.uri", "http://localhost:8090") - .config("spark.hadoop.fs.gravitino.client.metalake", "test_metalake") - .config("spark.hadoop.gcs-service-account-file", "/path/to/gcs-service-account-file.json") + .config("spark.hadoop.fs.gravitino.client.metalake", "metalake") + .config("spark.hadoop.gcs-service-account-file", "/path/to/service-account.json") .config("spark.driver.memory", "2g") .config("spark.driver.port", "2048") - .getOrCreate() + .getOrCreate()) data = [("Alice", 25), ("Bob", 30), ("Cathy", 45)] -columns = ["Name", "Age"] -spark_df = spark.createDataFrame(data, schema=columns) -gvfs_path = f"gvfs://fileset/{catalog_name}/{schema_name}/{fileset_name}/people" - -spark_df.coalesce(1).write - .mode("overwrite") - .option("header", "true") - .csv(gvfs_path) -``` - -If your Spark **without Hadoop environment**, you can use the following code snippet to access the fileset: - -```python -## Replace the following code snippet with the above code snippet with the same environment variables +spark_df = spark.createDataFrame(data, schema=["Name", "Age"]) +gvfs_path = "gvfs://fileset/gcs_catalog/gcs_schema/example_fileset/people" -os.environ["PYSPARK_SUBMIT_ARGS"] = "--jars /path/to/gravitino-gcp-bundle-{gravitino-version}.jar,/path/to/gravitino-filesystem-hadoop3-runtime-{gravitino-version}.jar, --master local[1] pyspark-shell" +spark_df.coalesce(1).write.mode("overwrite").option("header", "true").csv(gvfs_path) ``` -If Spark can't start with the above configuration (no Hadoop environment available and use bundle jar), you can try to set the jars to the classpath directly: +If Spark runs without a Hadoop environment, only the jar list changes: ```python -jars_path = ( - "/path/to/gravitino-gcp-bundle-{gravitino-version}.jar:" - "/path/to/gravitino-filesystem-hadoop3-runtime-{gravitino-version}.jar" -) - os.environ["PYSPARK_SUBMIT_ARGS"] = ( - f'--conf "spark.driver.extraClassPath={jars_path}" ' - f'--conf "spark.executor.extraClassPath={jars_path}" ' - '--master local[1] pyspark-shell' + "--jars /path/to/gravitino-gcp-bundle-${gravitino-version}.jar," + "/path/to/gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar " + "--master local[1] pyspark-shell" ) ``` -- [`gravitino-gcp-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-gcp-bundle): A "fat" JAR that includes `gravitino-gcp` functionality and all necessary dependencies like `gcs-connector` (hadoop3-2.2.22). Use this if your Spark environment doesn't have a pre-existing Hadoop setup. -- [`gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-filesystem-hadoop3-runtime): A "fat" JAR that bundles Gravitino's virtual filesystem client and includes the functionality of `gravitino-gcp`. It is required for accessing Gravitino filesets. -- [`gcs-connector-hadoop3-2.2.22-shaded.jar`](https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/download/v2.2.22/gcs-connector-hadoop3-2.2.22-shaded.jar): Standard Hadoop dependency for GCS access. If you are running in an existing Hadoop environment, you need to provide this JAR. -- [`gravitino-gcp-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-gcp): A "thin" JAR that only provides the GCP integration code. Its functionality is already included in the `gravitino-gcp-bundle` and `gravitino-filesystem-hadoop3-runtime` JARs, so you do not need to add it as a direct dependency unless you want to manage all Hadoop and GCP dependencies manually. - -Please choose the correct jar according to your environment. - :::note -In some Spark versions, a Hadoop environment is needed by the driver, adding the bundle jars with '--jars' may not work. If this is the case, you should add the jars to the spark CLASSPATH directly. +Some Spark versions need a Hadoop environment in the driver and do not pick up filesystem +implementations passed with `--jars`. If that happens, add the jars to the Spark classpath directly. ::: -### Access a Fileset Using the Hadoop Fs Command - -The following are examples of how to use the `hadoop fs` command to access the fileset in Hadoop 3.1.3: +### Hadoop fs command -1. Adding the following contents to the `${HADOOP_HOME}/etc/hadoop/core-site.xml` file: +1. Add the following to `${HADOOP_HOME}/etc/hadoop/core-site.xml`: ```xml - - fs.AbstractFileSystem.gvfs.impl - org.apache.gravitino.filesystem.hadoop.Gvfs - - - - fs.gvfs.impl - org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem - - - - fs.gravitino.server.uri - http://localhost:8090 - - - - fs.gravitino.client.metalake - test - - - - gcs-service-account-file - /path/your-service-account-file.json - + + fs.AbstractFileSystem.gvfs.impl + org.apache.gravitino.filesystem.hadoop.Gvfs + + + fs.gvfs.impl + org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem + + + fs.gravitino.server.uri + http://localhost:8090 + + + fs.gravitino.client.metalake + metalake + + + gcs-service-account-file + /path/to/service-account.json + ``` -2. Add the necessary jars to the Hadoop classpath. +2. Add these jars to the Hadoop classpath: -For GCS, you need to add `gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar` and [`gcs-connector-hadoop3-2.2.22-shaded.jar`](https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/download/v2.2.22/gcs-connector-hadoop3-2.2.22-shaded.jar) to Hadoop classpath. + - `gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`, from Maven Central. + - [`gcs-connector-hadoop3-2.2.22-shaded.jar`](https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/download/v2.2.22/gcs-connector-hadoop3-2.2.22-shaded.jar), published by Google and not part of the Apache Hadoop distribution. -3. Run the following command to access the fileset: +3. Access the fileset: ```shell -./${HADOOP_HOME}/bin/hadoop dfs -ls gvfs://fileset/gcs_catalog/gcs_schema/gcs_example -./${HADOOP_HOME}/bin/hadoop dfs -put /path/to/local/file gvfs://fileset/gcs_catalog/gcs_schema/gcs_example +${HADOOP_HOME}/bin/hadoop fs -ls gvfs://fileset/gcs_catalog/gcs_schema/example_fileset +${HADOOP_HOME}/bin/hadoop fs -put /path/to/local/file gvfs://fileset/gcs_catalog/gcs_schema/example_fileset ``` -### Access the Fileset with the GVFS Python Client - -To access fileset with GCS using the GVFS Python client, apart from [basic GVFS configurations](./how-to-use-gvfs.md#configuration-1), you need to add the following configurations: - -| Configuration item | Description | Default value | Required | -|----------------------------|--------------------------------------------|---------------|----------| -| `gcs_service_account_file` | The path of GCS service account JSON file. | (none) | Yes | - -:::note -If the catalog has enabled [credential vending](security/credential-vending.md), the properties above can be omitted. -::: - -Please install the `gravitino` package before running the following code: +### GVFS Python client ```bash pip install apache-gravitino==${GRAVITINO_VERSION} ``` +On top of the [base GVFS configuration](./how-to-use-gvfs.md#configuration-1), pass the Google Cloud Storage +properties in `options`, spelled with underscores. + ```python from gravitino import gvfs + options = { "cache_size": 20, "cache_expired_time": 3600, "auth_type": "simple", - "gcs_service_account_file": "path_of_gcs_service_account_file.json", + "gcs_service_account_file": "/path/to/service-account.json", } -fs = gvfs.GravitinoVirtualFileSystem(server_uri="http://localhost:8090", metalake_name="test_metalake", options=options) -fs.ls("gvfs://fileset/{catalog_name}/{schema_name}/{fileset_name}/") + +fs = gvfs.GravitinoVirtualFileSystem(server_uri="http://localhost:8090", + metalake_name="metalake", + options=options) +fs.ls("gvfs://fileset/gcs_catalog/gcs_schema/example_fileset/") ``` -### Access the Fileset with Pandas +### pandas -The following are examples of how to use the pandas library to access the GCS fileset +pandas reaches the same paths through `storage_options`. Use the `fs` instance from the preceding +GVFS example to discover the generated Spark part file. ```python import pandas as pd storage_options = { - "server_uri": "http://localhost:8090", - "metalake_name": "test", + "server_uri": "http://localhost:8090", + "metalake_name": "metalake", "options": { - "gcs_service_account_file": "path_of_gcs_service_account_file.json", + "gcs_service_account_file": "/path/to/service-account.json", } } -ds = pd.read_csv(f"gvfs://fileset/${catalog_name}/${schema_name}/${fileset_name}/people/part-00000-51d366e2-d5eb-448d-9109-32a96c8a14dc-c000.csv", - storage_options=storage_options) + +csv_path = next( + f"gvfs://{path}" + for path in fs.ls( + "gvfs://fileset/gcs_catalog/gcs_schema/example_fileset/people", + detail=False, + ) + if ( + path.rsplit("/", 1)[-1].startswith("part-") + and path.endswith(".csv") + ) +) +ds = pd.read_csv(csv_path, storage_options=storage_options) ds.head() ``` -For other use cases, refer to the [Gravitino Virtual File System](./how-to-use-gvfs.md) document. +For further use cases, see [Gravitino Virtual File System](./how-to-use-gvfs.md). -## Fileset with Credential Vending +## Credential Vending -Gravitino supports credential vending for GCS fileset. If the catalog has been [configured with credential](./security/credential-vending.md), you can access GCS fileset without providing authentication information like `gcs-service-account-file` in the properties. +With credential vending the catalog holds the Google Cloud Storage credentials and the Gravitino server hands +out a credential per request, so clients never hold cloud keys of their own. See +[Credential Vending](./security/credential-vending.md) for the general mechanism and +[GCS credentials](./security/credential-vending.md#gcs) for the properties +each provider takes. -### Create a GCS Fileset Catalog with Credential Vending +The supported provider is `gcs-token`, which vends a short-lived token. -Apart from configuration method in [create-gcs-fileset-catalog](#gcs-fileset-catalog-configuration), -properties needed by [gcs-credential](./security/credential-vending.md#gcs-credentials) should also -be set to enable credential vending for GCS fileset. Take `gcs-token` credential provider for example: +### Configure the catalog, schema, and fileset ```shell curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{ - "name": "gcs-catalog-with-token", + "name": "gcs_catalog_with_vending", "type": "FILESET", - "comment": "This is a GCS fileset catalog", + "comment": "A fileset catalog backed by Google Cloud Storage with credential vending", "properties": { "location": "gs://bucket/root", - "gcs-service-account-file": "path_of_gcs_service_account_file", + "gcs-service-account-file": "/path/to/service-account.json", "credential-providers": "gcs-token" } }' http://localhost:8090/api/metalakes/metalake/catalogs ``` -### Access a GCS Fileset with Credential Vending +Create the schema and fileset in the credential-vending catalog: -When the catalog is configured with credentials and client-side credential vending is enabled, -you can access GCS filesets directly using the GVFS Java/Python client or Spark without providing authentication details. +```shell +curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ +-H "Content-Type: application/json" -d '{ + "name": "gcs_schema", + "comment": "A schema in the Google Cloud Storage credential-vending catalog", + "properties": { + "location": "gs://bucket/root/schema" + } +}' http://localhost:8090/api/metalakes/metalake/catalogs/gcs_catalog_with_vending/schemas + +curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ +-H "Content-Type: application/json" -d '{ + "name": "example_fileset", + "comment": "This is an example fileset", + "type": "MANAGED", + "storageLocation": "gs://bucket/root/schema/example_fileset", + "properties": {} +}' http://localhost:8090/api/metalakes/metalake/catalogs/gcs_catalog_with_vending/schemas/gcs_schema/filesets +``` -GVFS Java client: +### Access without local credentials + +Enable vending on the client and drop the credential properties. ```java Configuration conf = new Configuration(); @@ -509,28 +516,34 @@ conf.setBoolean("fs.gravitino.enableCredentialVending", true); conf.set("fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs"); conf.set("fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem"); conf.set("fs.gravitino.server.uri", "http://localhost:8090"); -conf.set("fs.gravitino.client.metalake", "test_metalake"); +conf.set("fs.gravitino.client.metalake", "metalake"); // No need to set gcs-service-account-file -Path filesetPath = new Path("gvfs://fileset/gcs_test_catalog/test_schema/test_fileset/new_dir"); + +Path filesetPath = new Path( + "gvfs://fileset/gcs_catalog_with_vending/gcs_schema/example_fileset/new_dir"); FileSystem fs = filesetPath.getFileSystem(conf); fs.mkdirs(filesetPath); -... ``` -Spark: - ```python -spark = SparkSession.builder - .appName("gcs_fileset_test") +spark = (SparkSession.builder + .appName("gcs_fileset") .config("spark.hadoop.fs.gravitino.enableCredentialVending", "true") .config("spark.hadoop.fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs") .config("spark.hadoop.fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem") .config("spark.hadoop.fs.gravitino.server.uri", "http://localhost:8090") - .config("spark.hadoop.fs.gravitino.client.metalake", "test") + .config("spark.hadoop.fs.gravitino.client.metalake", "metalake") # No need to set gcs-service-account-file - .config("spark.driver.memory", "2g") - .config("spark.driver.port", "2048") - .getOrCreate() + .getOrCreate()) ``` -Python client and Hadoop command are similar to the above examples. +```python +options = { + "auth_type": "simple", + "enable_credential_vending": True, + # No need to set gcs-service-account-file +} +fs = gvfs.GravitinoVirtualFileSystem(server_uri="http://localhost:8090", + metalake_name="metalake", + options=options) +``` diff --git a/docs/fileset-catalog-with-oss.md b/docs/fileset-catalog-with-oss.md index 46a91f3f080..b6fa0a47666 100644 --- a/docs/fileset-catalog-with-oss.md +++ b/docs/fileset-catalog-with-oss.md @@ -1,63 +1,61 @@ --- title: "Fileset Catalog with OSS" slug: "/fileset-catalog-with-oss" -date: 2025-01-03 keyword: "Fileset catalog OSS" license: "This software is licensed under the Apache License version 2." --- ## Introduction -This document explains how to configure a Fileset catalog with Aliyun OSS (Object Storage Service) in Gravitino. +This page shows how to store fileset data in Alibaba Cloud OSS while Gravitino manages the metadata, +and how to read and write that data through the Gravitino Virtual File System (GVFS). -## Prerequisites +Everything on this page is specific to Alibaba Cloud OSS. The fileset model itself, the properties shared by +every storage backend, and the way properties are inherited from catalog to schema to fileset are +described in [Fileset Catalog](./fileset-catalog.md). + +The examples run in order and use the same names throughout: metalake `metalake`, catalog +`oss_catalog`, schema `oss_schema`, fileset `example_fileset`, and `http://localhost:8090` as the +server URL. Replace them with your own values. -To set up a Fileset catalog with OSS, follow these steps: +## Prerequisites 1. Download the [`gravitino-aliyun-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aliyun-bundle) file. -2. Place the downloaded file into the Gravitino Fileset catalog classpath at `${GRAVITINO_HOME}/catalogs/fileset/libs/`. -3. Start the Gravitino server by running the following command: +2. Place it in the fileset catalog classpath at `${GRAVITINO_HOME}/catalogs/fileset/libs/`. +3. Start the Gravitino server: ```bash -$ ${GRAVITINO_HOME}/bin/gravitino-server.sh start +${GRAVITINO_HOME}/bin/gravitino-server.sh start ``` -Once the server is up and running, you can proceed to configure the Fileset catalog with OSS. In the rest of this document we will use `http://localhost:8090` as the Gravitino server URL, replace with your actual server URL. - -## OSS Catalog Configuration - -### OSS Fileset Catalog Configuration - -In addition to the basic configurations mentioned in [Fileset-catalog-catalog-configuration](./fileset-catalog.md#catalog-properties), the following properties are required to configure a Fileset catalog with OSS: - -| Configuration item | Description | Default value | Required | -|-------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|----------| -| `filesystem-providers` | (deprecated) The file system providers to add. Set it to `oss` if it's a OSS fileset, or a comma separated string that contains `oss` like `oss,gs,s3` to support multiple kinds of fileset including `oss`. | (none) | Yes | -| `default-filesystem-provider` | (deprecated) The name default filesystem providers of this Fileset catalog if users do not specify the scheme in the URI. Default value is `builtin-local`, for OSS, if we set this value, we can omit the prefix 'oss://' in the location. | `builtin-local` | No | -| `oss-endpoint` | The endpoint of the Aliyun OSS. | (none) | Yes | -| `oss-access-key-id` | The access key of the Aliyun OSS. | (none) | Yes | -| `oss-secret-access-key` | The secret key of the Aliyun OSS. | (none) | Yes | -| `credential-providers` | The credential provider types, separated by comma, possible value can be `oss-token`, `oss-secret-key`. As the default authentication type is using AKSK as the above, this configuration can enable credential vending provided by Gravitino server and client will no longer need to provide authentication information like AKSK to access OSS by GVFS. Once it's set, more configuration items are needed to make it works, see [oss-credential-vending](security/credential-vending.md#oss-credentials) | (none) | No | +The catalog automatically loads the Alibaba Cloud OSS filesystem provider once the bundle jar is on the +classpath. The deprecated `filesystem-providers` and `default-filesystem-provider` catalog +properties do not need to be set. -:::note -`default-filesystem-provider` and `filesystem-providers` are deprecated. The fileset catalog automatically loads filesystem providers on the classpath, including buildin filesystem provider and cloud providers when the corresponding bundle jar is present (for example, `gravitino-aliyun-bundle`). -::: +## Alibaba Cloud OSS Properties -### Schema Configuration +These properties are needed in addition to the shared +[catalog properties](./fileset-catalog.md#catalog-properties). The same values are also needed by +the GVFS clients, so they are listed together here — note that the Python client spells them with +underscores while the catalog and the Java client use hyphens. -To create a schema, refer to [Schema configurations](./fileset-catalog.md#schema-properties). +| Catalog and Java client | Python client | Description | Required | +|-------------------------|-------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------| +| `oss-endpoint` | `oss_endpoint` | Endpoint of the Aliyun OSS service. | Yes | +| `oss-access-key-id` | `oss_access_key_id` | Access key of the Aliyun OSS service. | Yes | +| `oss-secret-access-key` | `oss_secret_access_key` | Secret key of the Aliyun OSS service. | Yes | +| `credential-providers` | (n/a) | The credential provider types, separated by comma. Possible values are `oss-token`, `oss-secret-key`. Setting it enables credential vending, so clients no longer need the credentials above. See [credential vending](./security/credential-vending.md#oss) for the extra properties each provider takes. | No | -### Fileset Configuration +Schema and fileset properties are documented on the shared page: see +[schema properties](./fileset-catalog.md#schema-properties) and +[fileset properties](./fileset-catalog.md#fileset-properties). -For instructions on how to create a fileset, refer to [Fileset configurations](./fileset-catalog.md#fileset-properties) for more details. +A fileset catalog stores its data under `location`, which for Alibaba Cloud OSS looks like +`oss://bucket/root`. ## Create the Catalog, Schema, and Fileset -This section will show you how to use the Fileset catalog with OSS in Gravitino, including detailed examples. - -### Step 1: Create a Fileset Catalog with OSS - -First, you need to create a Fileset catalog for OSS. The following examples demonstrate how to create a Fileset catalog with OSS: +### Step 1: Create the catalog @@ -65,14 +63,14 @@ First, you need to create a Fileset catalog for OSS. The following examples demo ```shell curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{ - "name": "test_catalog", + "name": "oss_catalog", "type": "FILESET", - "comment": "This is a OSS fileset catalog", + "comment": "A fileset catalog backed by Alibaba Cloud OSS", "properties": { "location": "oss://bucket/root", + "oss-endpoint": "http://oss-cn-hangzhou.aliyuncs.com", "oss-access-key-id": "access_key", - "oss-secret-access-key": "secret_key", - "oss-endpoint": "http://oss-cn-hangzhou.aliyuncs.com" + "oss-secret-access-key": "secret_key" } }' http://localhost:8090/api/metalakes/metalake/catalogs ``` @@ -86,46 +84,44 @@ GravitinoClient gravitinoClient = GravitinoClient .withMetalake("metalake") .build(); -Map ossProperties = ImmutableMap.builder() +Map catalogProperties = ImmutableMap.builder() .put("location", "oss://bucket/root") + .put("oss-endpoint", "http://oss-cn-hangzhou.aliyuncs.com") .put("oss-access-key-id", "access_key") .put("oss-secret-access-key", "secret_key") - .put("oss-endpoint", "http://oss-cn-hangzhou.aliyuncs.com") .build(); -Catalog ossCatalog = gravitinoClient.createCatalog("test_catalog", - Type.FILESET, - "This is a OSS fileset catalog", - ossProperties); -// ... - +Catalog catalog = gravitinoClient.createCatalog("oss_catalog", + Catalog.Type.FILESET, + "A fileset catalog backed by Alibaba Cloud OSS", + catalogProperties); ``` ```python -gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake") -oss_properties = { +gravitino_client: GravitinoClient = GravitinoClient( + uri="http://localhost:8090", metalake_name="metalake") + +catalog_properties = { "location": "oss://bucket/root", - "oss-access-key-id": "access_key" + "oss-endpoint": "http://oss-cn-hangzhou.aliyuncs.com", + "oss-access-key-id": "access_key", "oss-secret-access-key": "secret_key", - "oss-endpoint": "ossProperties" } -oss_catalog = gravitino_client.create_catalog(name="test_catalog", - catalog_type=Catalog.Type.FILESET, - provider=None, - comment="This is a OSS fileset catalog", - properties=oss_properties) +catalog = gravitino_client.create_catalog(name="oss_catalog", + catalog_type=Catalog.Type.FILESET, + provider=None, + comment="A fileset catalog backed by Alibaba Cloud OSS", + properties=catalog_properties) ``` -### Step 2: Create a Schema - -Once the Fileset catalog with OSS is created, you can create a schema inside that catalog. Below are examples of how to do this: +### Step 2: Create the schema @@ -133,50 +129,44 @@ Once the Fileset catalog with OSS is created, you can create a schema inside tha ```shell curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{ - "name": "test_schema", - "comment": "This is a OSS schema", + "name": "oss_schema", + "comment": "A schema in the Alibaba Cloud OSS fileset catalog", "properties": { "location": "oss://bucket/root/schema" } -}' http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas +}' http://localhost:8090/api/metalakes/metalake/catalogs/oss_catalog/schemas ``` ```java -Catalog catalog = gravitinoClient.loadCatalog("test_catalog"); - +Catalog catalog = gravitinoClient.loadCatalog("oss_catalog"); SupportsSchemas supportsSchemas = catalog.asSchemas(); Map schemaProperties = ImmutableMap.builder() .put("location", "oss://bucket/root/schema") .build(); -Schema schema = supportsSchemas.createSchema("test_schema", - "This is a OSS schema", - schemaProperties -); -// ... + +Schema schema = supportsSchemas.createSchema("oss_schema", + "A schema in the Alibaba Cloud OSS fileset catalog", + schemaProperties); ``` ```python -gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake") -catalog: Catalog = gravitino_client.load_catalog(name="test_catalog") -catalog.as_schemas().create_schema(name="test_schema", - comment="This is a OSS schema", +catalog: Catalog = gravitino_client.load_catalog(name="oss_catalog") +catalog.as_schemas().create_schema(name="oss_schema", + comment="A schema in the Alibaba Cloud OSS fileset catalog", properties={"location": "oss://bucket/root/schema"}) ``` - -### Step 3: Create a Fileset - -Now that the schema is created, you can create a fileset inside it. Here’s how: +### Step 3: Create the fileset @@ -191,346 +181,368 @@ curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ "properties": { "k1": "v1" } -}' http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas/test_schema/filesets +}' http://localhost:8090/api/metalakes/metalake/catalogs/oss_catalog/schemas/oss_schema/filesets ``` ```java -GravitinoClient gravitinoClient = GravitinoClient - .builder("http://localhost:8090") - .withMetalake("metalake") - .build(); - -Catalog catalog = gravitinoClient.loadCatalog("test_catalog"); +Catalog catalog = gravitinoClient.loadCatalog("oss_catalog"); FilesetCatalog filesetCatalog = catalog.asFilesetCatalog(); -Map propertiesMap = ImmutableMap.builder() - .put("k1", "v1") - .build(); +Map filesetProperties = ImmutableMap.builder() + .put("k1", "v1") + .build(); filesetCatalog.createFileset( - NameIdentifier.of("test_schema", "example_fileset"), + NameIdentifier.of("oss_schema", "example_fileset"), "This is an example fileset", Fileset.Type.MANAGED, "oss://bucket/root/schema/example_fileset", - propertiesMap, -); + filesetProperties); ``` ```python -gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake") - -catalog: Catalog = gravitino_client.load_catalog(name="test_catalog") -catalog.as_fileset_catalog().create_fileset(ident=NameIdentifier.of("test_schema", "example_fileset"), - type=Fileset.Type.MANAGED, - comment="This is an example fileset", - storage_location="oss://bucket/root/schema/example_fileset", - properties={"k1": "v1"}) +catalog: Catalog = gravitino_client.load_catalog(name="oss_catalog") +catalog.as_fileset_catalog().create_fileset( + ident=NameIdentifier.of("oss_schema", "example_fileset"), + type=Fileset.Type.MANAGED, + comment="This is an example fileset", + storage_location="oss://bucket/root/schema/example_fileset", + properties={"k1": "v1"}) ``` -## Access a Fileset with OSS +The fileset is now addressable as +`gvfs://fileset/oss_catalog/oss_schema/example_fileset` from any GVFS client. -### Access the Fileset with the GVFS Java Client +## Access the Fileset -To access fileset with OSS using the GVFS Java client, based on the [basic GVFS configurations](./how-to-use-gvfs.md#configuration-1), you need to add the following configurations: +### Java client jars -| Configuration item | Description | Default value | Required | -|-------------------------|-----------------------------------|---------------|----------| -| `oss-endpoint` | The endpoint of the Aliyun OSS. | (none) | Yes | -| `oss-access-key-id` | The access key of the Aliyun OSS. | (none) | Yes | -| `oss-secret-access-key` | The secret key of the Aliyun OSS. | (none) | Yes | +Every Java or Hadoop-based client needs `gravitino-filesystem-hadoop3-runtime`, which is published +on Maven Central, plus the Alibaba Cloud OSS filesystem implementation. Only the latter differs by +environment: -:::note -If the catalog has enabled [credential vending](security/credential-vending.md), the properties above can be omitted. More details can be found in [Fileset with credential vending](#fileset-with-credential-vending). -::: +| Environment | Jar providing the Alibaba Cloud OSS filesystem | +|------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| No Hadoop installed | [`gravitino-aliyun-bundle`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aliyun-bundle), a fat jar bundling the Alibaba Cloud OSS filesystem implementation and its dependencies | +| Hadoop already present | `hadoop-aliyun-${hadoop-version}.jar`, `aliyun-sdk-oss-3.13.0.jar` and `jdom2-2.0.6.jar`, shipped with Hadoop under `${HADOOP_HOME}/share/hadoop/tools/lib` | -```java -Configuration conf = new Configuration(); -conf.set("fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs"); -conf.set("fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem"); -conf.set("fs.gravitino.server.uri", "http://localhost:8090"); -conf.set("fs.gravitino.client.metalake", "test_metalake"); -conf.set("oss-endpoint", "http://localhost:8090"); -conf.set("oss-access-key-id", "minio"); -conf.set("oss-secret-access-key", "minio123"); -Path filesetPath = new Path("gvfs://fileset/test_catalog/test_schema/test_fileset/new_dir"); -FileSystem fs = filesetPath.getFileSystem(conf); -fs.mkdirs(filesetPath); -... -``` +The artifacts in full: + +- [`gravitino-aliyun-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aliyun-bundle): + a "fat" jar that includes the `gravitino-aliyun` functionality together with every dependency it needs, + such as `hadoop-aliyun` and `aliyun-sdk-oss`. Use it when the environment has no pre-existing Hadoop setup. +- [`gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-filesystem-hadoop3-runtime): + a "fat" jar that bundles the Gravitino virtual filesystem client and already includes the + `gravitino-aliyun` functionality. Java and Hadoop-based clients require it to access Gravitino + filesets. +- `hadoop-aliyun-${hadoop-version}.jar`, `aliyun-sdk-oss-3.13.0.jar` and `jdom2-2.0.6.jar`: the + standard Hadoop dependencies for Alibaba Cloud OSS access, shipped with Hadoop under + `${HADOOP_HOME}/share/hadoop/tools/lib`. Supply them yourself when running inside an existing + Hadoop environment. +- [`gravitino-aliyun-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aliyun): + a "thin" jar carrying only the Aliyun integration code. It is already contained in both jars + above, so it is not needed as a direct dependency unless you prefer to manage all Hadoop and + Aliyun dependencies yourself. -Similar to Spark configurations, you need to add OSS (bundle) jars to the classpath according to your environment. -If your wants to custom your hadoop version or there is already a hadoop version in your project, you can add the following dependencies to your `pom.xml`: +```xml + + + org.apache.gravitino + gravitino-aliyun-bundle + ${GRAVITINO_VERSION} + + + org.apache.gravitino + gravitino-filesystem-hadoop3-runtime + ${GRAVITINO_VERSION} + +``` ```xml - - org.apache.hadoop - hadoop-common - ${HADOOP_VERSION} - - - - org.apache.hadoop - hadoop-aliyun - ${HADOOP_VERSION} - - - - org.apache.gravitino - gravitino-filesystem-hadoop3-runtime - ${GRAVITINO_VERSION} - + + + org.apache.hadoop + hadoop-common + ${HADOOP_VERSION} + + + org.apache.hadoop + hadoop-aliyun + ${HADOOP_VERSION} + + + org.apache.gravitino + gravitino-filesystem-hadoop3-runtime + ${GRAVITINO_VERSION} + ``` :::note -The `gravitino-aliyun` JAR is no longer required, as it is now included in the `gravitino-filesystem-hadoop3-runtime` JAR. +The thin `gravitino-aliyun` jar is not needed. Its functionality is already included in both +`gravitino-aliyun-bundle` and `gravitino-filesystem-hadoop3-runtime`. ::: -Or use the bundle jar with Hadoop environment if there is no Hadoop environment: +### GVFS Java client -```xml - - org.apache.gravitino - gravitino-aliyun-bundle - ${GRAVITINO_VERSION} - - - - org.apache.gravitino - gravitino-filesystem-hadoop3-runtime - ${GRAVITINO_VERSION} - -``` +On top of the [base GVFS configuration](./how-to-use-gvfs.md#configuration), set the Alibaba Cloud OSS +properties from the table above. + +```java +Configuration conf = new Configuration(); +conf.set("fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs"); +conf.set("fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem"); +conf.set("fs.gravitino.server.uri", "http://localhost:8090"); +conf.set("fs.gravitino.client.metalake", "metalake"); +conf.set("oss-endpoint", "http://oss-cn-hangzhou.aliyuncs.com"); +conf.set("oss-access-key-id", "access_key"); +conf.set("oss-secret-access-key", "secret_key"); -### Access the Fileset with Spark +Path filesetPath = new Path("gvfs://fileset/oss_catalog/oss_schema/example_fileset/new_dir"); +FileSystem fs = filesetPath.getFileSystem(conf); +fs.mkdirs(filesetPath); +``` -The following code snippet shows how to use **PySpark 3.5.0 with Hadoop environment(Hadoop 3.3.4)** to access the fileset: +### Apache Spark -Before running the following code, you need to install required packages: +The example below uses PySpark 3.5.0 in an environment that already has Hadoop 3.3.4. ```bash pip install pyspark==3.5.0 pip install apache-gravitino==${GRAVITINO_VERSION} ``` -Then you can run the following code: ```python -from pyspark.sql import SparkSession import os +from pyspark.sql import SparkSession -gravitino_url = "http://localhost:8090" -metalake_name = "test" - -catalog_name = "your_oss_catalog" -schema_name = "your_oss_schema" -fileset_name = "your_oss_fileset" - -# JDK8 as follows, JDK17 will be slightly different, you need to add '--conf \"spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED\" --conf \"spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED\"' to the submit args. +# On JDK 17, also add: +# --conf "spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" +# --conf "spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" os.environ["PYSPARK_SUBMIT_ARGS"] = ( - "--jars " - "/path/to/gravitino-filesystem-hadoop3-runtime-{gravitino-version}.jar," - "/path/to/aliyun-sdk-oss-3.13.0.jar," + "--jars /path/to/gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar," "/path/to/hadoop-aliyun-3.3.4.jar," - "/path/to/jdom2-2.0.6 " + "/path/to/aliyun-sdk-oss-3.13.0.jar," + "/path/to/jdom2-2.0.6.jar " "--master local[1] pyspark-shell" ) -spark = SparkSession.builder - .appName("oss_fileset_test") + +spark = (SparkSession.builder + .appName("oss_fileset") .config("spark.hadoop.fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs") .config("spark.hadoop.fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem") - .config("spark.hadoop.fs.gravitino.server.uri", "${_URL}") - .config("spark.hadoop.fs.gravitino.client.metalake", "test") - .config("spark.hadoop.oss-access-key-id", os.environ["OSS_ACCESS_KEY_ID"]) - .config("spark.hadoop.oss-secret-access-key", os.environ["OSS_SECRET_ACCESS_KEY"]) + .config("spark.hadoop.fs.gravitino.server.uri", "http://localhost:8090") + .config("spark.hadoop.fs.gravitino.client.metalake", "metalake") .config("spark.hadoop.oss-endpoint", "http://oss-cn-hangzhou.aliyuncs.com") + .config("spark.hadoop.oss-access-key-id", "access_key") + .config("spark.hadoop.oss-secret-access-key", "secret_key") .config("spark.driver.memory", "2g") .config("spark.driver.port", "2048") - .getOrCreate() + .getOrCreate()) data = [("Alice", 25), ("Bob", 30), ("Cathy", 45)] -columns = ["Name", "Age"] -spark_df = spark.createDataFrame(data, schema=columns) -gvfs_path = f"gvfs://fileset/{catalog_name}/{schema_name}/{fileset_name}/people" - -spark_df.coalesce(1).write - .mode("overwrite") - .option("header", "true") - .csv(gvfs_path) +spark_df = spark.createDataFrame(data, schema=["Name", "Age"]) +gvfs_path = "gvfs://fileset/oss_catalog/oss_schema/example_fileset/people" + +spark_df.coalesce(1).write.mode("overwrite").option("header", "true").csv(gvfs_path) ``` -If your Spark **without Hadoop environment**, you can use the following code snippet to access the fileset: +If Spark runs without a Hadoop environment, only the jar list changes: ```python -## Replace the following code snippet with the above code snippet with the same environment variables - -os.environ["PYSPARK_SUBMIT_ARGS"] = "--jars /path/to/gravitino-aliyun-bundle-{gravitino-version}.jar,/path/to/gravitino-filesystem-hadoop3-runtime-{gravitino-version}.jar, --master local[1] pyspark-shell" +os.environ["PYSPARK_SUBMIT_ARGS"] = ( + "--jars /path/to/gravitino-aliyun-bundle-${gravitino-version}.jar," + "/path/to/gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar " + "--master local[1] pyspark-shell" +) ``` -- [`gravitino-aliyun-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aliyun-bundle): A "fat" JAR that includes `gravitino-aliyun` functionality and all necessary dependencies like `hadoop-aliyun` (3.3.1) and `aliyun-sdk-oss`. Use this if your Spark environment doesn't have a pre-existing Hadoop setup. -- [`gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-filesystem-hadoop3-runtime): A "fat" JAR that bundles Gravitino's virtual filesystem client and includes the functionality of `gravitino-aliyun`. It is required for accessing Gravitino filesets. -- `hadoop-aliyun-3.3.4.jar`, `jdom2-2.0.6.jar`, and `aliyun-sdk-oss-3.13.0.jar`: Standard Hadoop dependencies for OSS access. If you are running in an existing Hadoop environment, you need to provide these JARs. They are typically located in the `${HADOOP_HOME}/share/hadoop/tools/lib` directory. -- [`gravitino-aliyun-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aliyun): A "thin" JAR that only provides the Aliyun integration code. Its functionality is already included in the `gravitino-aliyun-bundle` and `gravitino-filesystem-hadoop3-runtime` JARs, so you do not need to add it as a direct dependency unless you want to manage all Hadoop and Aliyun dependencies manually. - -Please choose the correct jar according to your environment. - :::note -In some Spark versions, a Hadoop environment is needed by the driver, adding the bundle jars with '--jars' may not work. If this is the case, you should add the jars to the spark CLASSPATH directly. +Some Spark versions need a Hadoop environment in the driver and do not pick up filesystem +implementations passed with `--jars`. If that happens, add the jars to the Spark classpath directly. ::: -### Access a Fileset Using the Hadoop Fs Command - -The following are examples of how to use the `hadoop fs` command to access the fileset in Hadoop 3.1.3: +### Hadoop fs command -1. Adding the following contents to the `${HADOOP_HOME}/etc/hadoop/core-site.xml` file: +1. Add the following to `${HADOOP_HOME}/etc/hadoop/core-site.xml`: ```xml - - fs.AbstractFileSystem.gvfs.impl - org.apache.gravitino.filesystem.hadoop.Gvfs - - - - fs.gvfs.impl - org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem - - - - fs.gravitino.server.uri - http://localhost:8090 - - - - fs.gravitino.client.metalake - test - - - - oss-endpoint - http://oss-cn-hangzhou.aliyuncs.com - - - - oss-access-key-id - access-key - - - + + fs.AbstractFileSystem.gvfs.impl + org.apache.gravitino.filesystem.hadoop.Gvfs + + + fs.gvfs.impl + org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem + + + fs.gravitino.server.uri + http://localhost:8090 + + + fs.gravitino.client.metalake + metalake + + + oss-endpoint + http://oss-cn-hangzhou.aliyuncs.com + + + oss-access-key-id + access_key + + oss-secret-access-key - secret-key - + secret_key + ``` -2. Add the necessary jars to the Hadoop classpath. +2. Add these jars to the Hadoop classpath: -For OSS, you need to add `gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar` and `hadoop-aliyun-${hadoop-version}.jar` located at `${HADOOP_HOME}/share/hadoop/tools/lib/` to Hadoop classpath. + - `gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`, from Maven Central. + - `hadoop-aliyun-${hadoop-version}.jar`, `aliyun-sdk-oss-3.13.0.jar` and `jdom2-2.0.6.jar`, shipped with Hadoop under `${HADOOP_HOME}/share/hadoop/tools/lib`. -3. Run the following command to access the fileset: +3. Access the fileset: ```shell -./${HADOOP_HOME}/bin/hadoop dfs -ls gvfs://fileset/oss_catalog/oss_schema/oss_fileset -./${HADOOP_HOME}/bin/hadoop dfs -put /path/to/local/file gvfs://fileset/oss_catalog/schema/oss_fileset +${HADOOP_HOME}/bin/hadoop fs -ls gvfs://fileset/oss_catalog/oss_schema/example_fileset +${HADOOP_HOME}/bin/hadoop fs -put /path/to/local/file gvfs://fileset/oss_catalog/oss_schema/example_fileset ``` -### Access the Fileset with the GVFS Python Client - -To access fileset with OSS using the GVFS Python client, apart from [basic GVFS configurations](./how-to-use-gvfs.md#configuration-1), you need to add the following configurations: - -| Configuration item | Description | Default value | Required | -|-------------------------|-----------------------------------|---------------|----------| -| `oss_endpoint` | The endpoint of the Aliyun OSS. | (none) | Yes | -| `oss_access_key_id` | The access key of the Aliyun OSS. | (none) | Yes | -| `oss_secret_access_key` | The secret key of the Aliyun OSS. | (none) | Yes | - -:::note -If the catalog has enabled [credential vending](security/credential-vending.md), the properties above can be omitted. -::: - -Please install the `gravitino` package before running the following code: +### GVFS Python client ```bash pip install apache-gravitino==${GRAVITINO_VERSION} ``` +On top of the [base GVFS configuration](./how-to-use-gvfs.md#configuration-1), pass the Alibaba Cloud OSS +properties in `options`, spelled with underscores. + ```python from gravitino import gvfs + options = { "cache_size": 20, "cache_expired_time": 3600, "auth_type": "simple", - "oss_endpoint": "http://localhost:8090", - "oss_access_key_id": "minio", - "oss_secret_access_key": "minio123" + "oss_endpoint": "http://oss-cn-hangzhou.aliyuncs.com", + "oss_access_key_id": "access_key", + "oss_secret_access_key": "secret_key", } -fs = gvfs.GravitinoVirtualFileSystem(server_uri="http://localhost:8090", metalake_name="test_metalake", options=options) -fs.ls("gvfs://fileset/{catalog_name}/{schema_name}/{fileset_name}/") +fs = gvfs.GravitinoVirtualFileSystem(server_uri="http://localhost:8090", + metalake_name="metalake", + options=options) +fs.ls("gvfs://fileset/oss_catalog/oss_schema/example_fileset/") ``` +### pandas -### Access the Fileset with Pandas - -The following are examples of how to use the pandas library to access the OSS fileset +pandas reaches the same paths through `storage_options`. Use the `fs` instance from the preceding +GVFS example to discover the generated Spark part file. ```python import pandas as pd storage_options = { - "server_uri": "http://mini.io:9000", - "metalake_name": "test", + "server_uri": "http://localhost:8090", + "metalake_name": "metalake", "options": { + "oss_endpoint": "http://oss-cn-hangzhou.aliyuncs.com", "oss_access_key_id": "access_key", "oss_secret_access_key": "secret_key", - "oss_endpoint": "http://oss-cn-hangzhou.aliyuncs.com" } } -ds = pd.read_csv(f"gvfs://fileset/${catalog_name}/${schema_name}/${fileset_name}/people/part-00000-51d366e2-d5eb-448d-9109-32a96c8a14dc-c000.csv", - storage_options=storage_options) + +csv_path = next( + f"gvfs://{path}" + for path in fs.ls( + "gvfs://fileset/oss_catalog/oss_schema/example_fileset/people", + detail=False, + ) + if ( + path.rsplit("/", 1)[-1].startswith("part-") + and path.endswith(".csv") + ) +) +ds = pd.read_csv(csv_path, storage_options=storage_options) ds.head() ``` -For other use cases, refer to the [Gravitino Virtual File System](./how-to-use-gvfs.md) document. -## Fileset with Credential Vending +For further use cases, see [Gravitino Virtual File System](./how-to-use-gvfs.md). -Gravitino supports credential vending for OSS fileset. If the catalog has been [configured with credential](./security/credential-vending.md), you can access OSS fileset without providing authentication information like `oss-access-key-id` and `oss-secret-access-key` in the properties. +## Credential Vending -### Create an OSS Fileset Catalog with Credential Vending +With credential vending the catalog holds the Alibaba Cloud OSS credentials and the Gravitino server hands +out a credential per request, so clients never hold cloud keys of their own. See +[Credential Vending](./security/credential-vending.md) for the general mechanism and +[OSS credentials](./security/credential-vending.md#oss) for the properties +each provider takes. -Apart from configuration method in [create-oss-fileset-catalog](#oss-fileset-catalog-configuration), -properties needed by [oss-credential](./security/credential-vending.md#oss-credentials) -should also be set to enable credential vending for OSS fileset. Take `oss-token` credential provider for example: +The supported providers are `oss-token`, which vends a short-lived STS token, and +`oss-secret-key`, which vends the static access key configured on the catalog. The example below uses +`oss-token`. + +### Configure the catalog, schema, and fileset ```shell curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{ - "name": "oss-catalog-with-token", + "name": "oss_catalog_with_vending", "type": "FILESET", - "comment": "This is a OSS fileset catalog", + "comment": "A fileset catalog backed by Alibaba Cloud OSS with credential vending", "properties": { "location": "oss://bucket/root", + "oss-endpoint": "http://oss-cn-hangzhou.aliyuncs.com", "oss-access-key-id": "access_key", "oss-secret-access-key": "secret_key", - "oss-endpoint": "http://oss-cn-hangzhou.aliyuncs.com", "credential-providers": "oss-token", - "oss-region":"oss-cn-hangzhou", - "oss-role-arn":"The ARN of the role to access the OSS data" + "oss-region": "oss-cn-hangzhou", + "oss-role-arn": "The ARN of the role that grants access to the OSS data" } }' http://localhost:8090/api/metalakes/metalake/catalogs ``` -### Access an OSS Fileset with Credential Vending +Create the schema and fileset in the credential-vending catalog: -When the catalog is configured with credentials and client-side credential vending is enabled, -you can access OSS filesets directly using the GVFS Java/Python client or Spark without providing authentication details. +```shell +curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ +-H "Content-Type: application/json" -d '{ + "name": "oss_schema", + "comment": "A schema in the Alibaba Cloud OSS credential-vending catalog", + "properties": { + "location": "oss://bucket/root/schema" + } +}' http://localhost:8090/api/metalakes/metalake/catalogs/oss_catalog_with_vending/schemas + +curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ +-H "Content-Type: application/json" -d '{ + "name": "example_fileset", + "comment": "This is an example fileset", + "type": "MANAGED", + "storageLocation": "oss://bucket/root/schema/example_fileset", + "properties": {} +}' http://localhost:8090/api/metalakes/metalake/catalogs/oss_catalog_with_vending/schemas/oss_schema/filesets +``` -GVFS Java client: +The `oss-token` provider needs two more catalog properties. + +| Property Name | Description | +|----------------|-----------------------------------------------------| +| `oss-region` | Region of the bucket, for example `oss-cn-hangzhou` | +| `oss-role-arn` | ARN of the role that grants access to the data | + +### Access without local credentials + +Enable vending on the client and drop the credential properties. ```java Configuration conf = new Configuration(); @@ -538,29 +550,34 @@ conf.setBoolean("fs.gravitino.enableCredentialVending", true); conf.set("fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs"); conf.set("fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem"); conf.set("fs.gravitino.server.uri", "http://localhost:8090"); -conf.set("fs.gravitino.client.metalake", "test_metalake"); -// No need to set oss-access-key-id and oss-secret-access-key -Path filesetPath = new Path("gvfs://fileset/oss_test_catalog/test_schema/test_fileset/new_dir"); +conf.set("fs.gravitino.client.metalake", "metalake"); +// No need to set oss-access-key-id or oss-secret-access-key + +Path filesetPath = new Path( + "gvfs://fileset/oss_catalog_with_vending/oss_schema/example_fileset/new_dir"); FileSystem fs = filesetPath.getFileSystem(conf); fs.mkdirs(filesetPath); -... ``` -Spark: - ```python -spark = SparkSession.builder - .appName("oss_fileset_test") +spark = (SparkSession.builder + .appName("oss_fileset") .config("spark.hadoop.fs.gravitino.enableCredentialVending", "true") .config("spark.hadoop.fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs") .config("spark.hadoop.fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem") .config("spark.hadoop.fs.gravitino.server.uri", "http://localhost:8090") - .config("spark.hadoop.fs.gravitino.client.metalake", "test") - # No need to set oss-access-key-id and oss-secret-access-key - .config("spark.driver.memory", "2g") - .config("spark.driver.port", "2048") - .getOrCreate() + .config("spark.hadoop.fs.gravitino.client.metalake", "metalake") + # No need to set oss-access-key-id or oss-secret-access-key + .getOrCreate()) ``` -Python client and Hadoop command are similar to the above examples. - +```python +options = { + "auth_type": "simple", + "enable_credential_vending": True, + # No need to set oss-access-key-id or oss-secret-access-key +} +fs = gvfs.GravitinoVirtualFileSystem(server_uri="http://localhost:8090", + metalake_name="metalake", + options=options) +``` diff --git a/docs/fileset-catalog-with-s3.md b/docs/fileset-catalog-with-s3.md index 8c9e5adf184..bc523ce2fac 100644 --- a/docs/fileset-catalog-with-s3.md +++ b/docs/fileset-catalog-with-s3.md @@ -1,63 +1,74 @@ --- title: "Fileset Catalog with S3" slug: "/fileset-catalog-with-s3" -date: 2025-01-03 keyword: "Fileset catalog S3" license: "This software is licensed under the Apache License version 2." --- ## Introduction -This document explains how to configure a Fileset catalog with S3 in Gravitino. +This page shows how to store fileset data in Amazon S3 while Gravitino manages the metadata, +and how to read and write that data through the Gravitino Virtual File System (GVFS). -## Prerequisites +Everything on this page is specific to Amazon S3. The fileset model itself, the properties shared by +every storage backend, and the way properties are inherited from catalog to schema to fileset are +described in [Fileset Catalog](./fileset-catalog.md). + +The examples run in order and use the same names throughout: metalake `metalake`, catalog +`s3_catalog`, schema `s3_schema`, fileset `example_fileset`, and `http://localhost:8090` as the +server URL. Replace them with your own values. -To create a Fileset catalog with S3, follow these steps: +## Prerequisites 1. Download the [`gravitino-aws-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aws-bundle) file. -2. Place this file in the Gravitino Fileset catalog classpath at `${GRAVITINO_HOME}/catalogs/fileset/libs/`. -3. Start the Gravitino server using the following command: +2. Place it in the fileset catalog classpath at `${GRAVITINO_HOME}/catalogs/fileset/libs/`. +3. Start the Gravitino server: ```bash -$ ${GRAVITINO_HOME}/bin/gravitino-server.sh start +${GRAVITINO_HOME}/bin/gravitino-server.sh start ``` -Once the server is up and running, you can proceed to configure the Fileset catalog with S3. In the rest of this document we will use `http://localhost:8090` as the Gravitino server URL, replace with your actual server URL. - -## S3 Catalog Configuration +The catalog automatically loads the Amazon S3 filesystem provider once the bundle jar is on the +classpath. The deprecated `filesystem-providers` and `default-filesystem-provider` catalog +properties do not need to be set. -### S3 Fileset Catalog Configuration +## Amazon S3 Properties -In addition to the basic configurations mentioned in [Fileset-catalog-catalog-configuration](./fileset-catalog.md#catalog-properties), the following properties are necessary to configure a Fileset catalog with S3: +These properties are needed in addition to the shared +[catalog properties](./fileset-catalog.md#catalog-properties). The same values are also needed by +the GVFS clients, so they are listed together here — note that the Python client spells them with +underscores while the catalog and the Java client use hyphens. -| Configuration item | Description | Default value | Required | -|-------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|----------| -| `filesystem-providers` | (deprecated) The file system providers to add. Set it to `s3` if it's a S3 fileset, or a comma separated string that contains `s3` like `gs,s3` to support multiple kinds of fileset including `s3`. | (none) | Yes | -| `default-filesystem-provider` | (deprecated) The name default filesystem providers of this Fileset catalog if users do not specify the scheme in the URI. Default value is `builtin-local`, for S3, if we set this value, we can omit the prefix 's3a://' in the location. | `builtin-local` | No | -| `s3-endpoint` | The endpoint of the AWS S3. | (none) | Yes | -| `s3-access-key-id` | The access key of the AWS S3. | (none) | Yes | -| `s3-secret-access-key` | The secret key of the AWS S3. | (none) | Yes | -| `credential-providers` | The credential provider types, separated by comma, possible value can be `s3-token`, `s3-secret-key`. As the default authentication type is using AKSK as the above, this configuration can enable credential vending provided by Gravitino server and client will no longer need to provide authentication information like AKSK to access S3 by GVFS. Once it's set, more configuration items are needed to make it works, see [s3-credential-vending](security/credential-vending.md#s3-credentials) | (none) | No | +| Catalog and Java client | Python client | Description | Required | +|-------------------------|------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------| +| `s3-endpoint` | `s3_endpoint` | Endpoint of the S3 service. S3-compatible storage such as MinIO always needs it. | Yes, except for the Python client against AWS S3 | +| `s3-access-key-id` | `s3_access_key_id` | Access key of the S3 service. | Yes | +| `s3-secret-access-key` | `s3_secret_access_key` | Secret key of the S3 service. | Yes | +| `credential-providers` | (n/a) | The credential provider types, separated by comma. Possible values are `s3-token`, `s3-secret-key`, `aws-irsa`. Setting it enables credential vending, so clients no longer need the credentials above. See [credential vending](./security/credential-vending.md#s3) for the extra properties each provider takes. | No | :::note -`default-filesystem-provider` and `filesystem-providers` are deprecated. The fileset catalog automatically loads filesystem providers on the classpath, including buildin filesystem provider and cloud providers when the corresponding bundle jar is present (for example, `gravitino-aws-bundle`). +- The location must start with `s3a://`, not `s3://`. The `hadoop-aws` library does not support the + `s3://` scheme. +- For MinIO and other S3-compatible services, set `s3-endpoint` to that service. If it requires + path-style access, add `gravitino.bypass.fs.s3a.path.style.access=true` to + `${GRAVITINO_HOME}/catalogs/fileset/conf/fileset.conf` for server-side operations. Also set + `s3-path-style-access=true` on a GVFS Java client, or + `spark.hadoop.s3-path-style-access=true` in Spark. For the Python GVFS client, pass + `config_kwargs={"s3": {"addressing_style": "path"}}` to + `GravitinoVirtualFileSystem`; for pandas, add the same `config_kwargs` entry at the top level + of `storage_options`. ::: -### Schema Configuration +Schema and fileset properties are documented on the shared page: see +[schema properties](./fileset-catalog.md#schema-properties) and +[fileset properties](./fileset-catalog.md#fileset-properties). -To learn how to create a schema, refer to [Schema configurations](./fileset-catalog.md#schema-properties). - -### Fileset Configuration - -For more details on creating a fileset, Refer to [Fileset configurations](./fileset-catalog.md#fileset-properties). +A fileset catalog stores its data under `location`, which for Amazon S3 looks like +`s3a://bucket/root`. ## Create the Catalog, Schema, and Fileset -This section demonstrates how to use the Fileset catalog with S3 in Gravitino, with a complete example. - -### Step 1: Create a Fileset Catalog with S3 - -First of all, you need to create a Fileset catalog with S3. The following example shows how to create a Fileset catalog with S3: +### Step 1: Create the catalog @@ -65,14 +76,14 @@ First of all, you need to create a Fileset catalog with S3. The following exampl ```shell curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{ - "name": "test_catalog", + "name": "s3_catalog", "type": "FILESET", - "comment": "This is a S3 fileset catalog", + "comment": "A fileset catalog backed by Amazon S3", "properties": { "location": "s3a://bucket/root", + "s3-endpoint": "http://s3.ap-northeast-1.amazonaws.com", "s3-access-key-id": "access_key", - "s3-secret-access-key": "secret_key", - "s3-endpoint": "http://s3.ap-northeast-1.amazonaws.com" + "s3-secret-access-key": "secret_key" } }' http://localhost:8090/api/metalakes/metalake/catalogs ``` @@ -86,56 +97,44 @@ GravitinoClient gravitinoClient = GravitinoClient .withMetalake("metalake") .build(); -Map s3Properties = ImmutableMap.builder() +Map catalogProperties = ImmutableMap.builder() .put("location", "s3a://bucket/root") + .put("s3-endpoint", "http://s3.ap-northeast-1.amazonaws.com") .put("s3-access-key-id", "access_key") .put("s3-secret-access-key", "secret_key") - .put("s3-endpoint", "http://s3.ap-northeast-1.amazonaws.com") .build(); -Catalog s3Catalog = gravitinoClient.createCatalog("test_catalog", - Type.FILESET, - "This is a S3 fileset catalog", - s3Properties); -// ... - +Catalog catalog = gravitinoClient.createCatalog("s3_catalog", + Catalog.Type.FILESET, + "A fileset catalog backed by Amazon S3", + catalogProperties); ``` ```python -gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake") -s3_properties = { +gravitino_client: GravitinoClient = GravitinoClient( + uri="http://localhost:8090", metalake_name="metalake") + +catalog_properties = { "location": "s3a://bucket/root", - "s3-access-key-id": "access_key" + "s3-endpoint": "http://s3.ap-northeast-1.amazonaws.com", + "s3-access-key-id": "access_key", "s3-secret-access-key": "secret_key", - "s3-endpoint": "http://s3.ap-northeast-1.amazonaws.com" } -s3_catalog = gravitino_client.create_catalog(name="test_catalog", - catalog_type=Catalog.Type.FILESET, - provider=None, - comment="This is a S3 fileset catalog", - properties=s3_properties) +catalog = gravitino_client.create_catalog(name="s3_catalog", + catalog_type=Catalog.Type.FILESET, + provider=None, + comment="A fileset catalog backed by Amazon S3", + properties=catalog_properties) ``` -:::note -- When using S3, ensure that the location value starts with s3a:// (not s3://) for AWS S3. For example, use s3a://bucket/root, as the s3:// format is not supported by the hadoop-aws library. -- When using MinIO or other S3-compatible storage services, make sure to set the `s3-endpoint` property to the appropriate endpoint URL. -- When using MinIO or other S3-compatible storage services, you may need to set additional properties such as `s3-path-style-access` to `true` depending on the storage service requirements. You can do this in Gravitino's `fileset.conf` file with the "gravitino.bypass." prefix: -```bash -$ cat $GRAVITINO_HOME/catalogs/fileset/conf/fileset.conf -gravitino.bypass.fs.s3a.path.style.access=true -``` -::: - -### Step 2: Create a Schema - -Once your Fileset catalog with S3 is created, you can create a schema under the catalog. Here are examples of how to do that: +### Step 2: Create the schema @@ -143,49 +142,44 @@ Once your Fileset catalog with S3 is created, you can create a schema under the ```shell curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{ - "name": "test_schema", - "comment": "This is a S3 schema", + "name": "s3_schema", + "comment": "A schema in the Amazon S3 fileset catalog", "properties": { "location": "s3a://bucket/root/schema" } -}' http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas +}' http://localhost:8090/api/metalakes/metalake/catalogs/s3_catalog/schemas ``` ```java -Catalog catalog = gravitinoClient.loadCatalog("hive_catalog"); - +Catalog catalog = gravitinoClient.loadCatalog("s3_catalog"); SupportsSchemas supportsSchemas = catalog.asSchemas(); Map schemaProperties = ImmutableMap.builder() .put("location", "s3a://bucket/root/schema") .build(); -Schema schema = supportsSchemas.createSchema("test_schema", - "This is a S3 schema", - schemaProperties -); -// ... + +Schema schema = supportsSchemas.createSchema("s3_schema", + "A schema in the Amazon S3 fileset catalog", + schemaProperties); ``` ```python -gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake") -catalog: Catalog = gravitino_client.load_catalog(name="test_catalog") -catalog.as_schemas().create_schema(name="test_schema", - comment="This is a S3 schema", +catalog: Catalog = gravitino_client.load_catalog(name="s3_catalog") +catalog.as_schemas().create_schema(name="s3_schema", + comment="A schema in the Amazon S3 fileset catalog", properties={"location": "s3a://bucket/root/schema"}) ``` -### Step 3: Create a Fileset - -After creating the schema, you can create a fileset. Here are examples for creating a fileset: +### Step 3: Create the fileset @@ -200,339 +194,368 @@ curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ "properties": { "k1": "v1" } -}' http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas/test_schema/filesets +}' http://localhost:8090/api/metalakes/metalake/catalogs/s3_catalog/schemas/s3_schema/filesets ``` ```java -GravitinoClient gravitinoClient = GravitinoClient - .builder("http://localhost:8090") - .withMetalake("metalake") - .build(); - -Catalog catalog = gravitinoClient.loadCatalog("test_catalog"); +Catalog catalog = gravitinoClient.loadCatalog("s3_catalog"); FilesetCatalog filesetCatalog = catalog.asFilesetCatalog(); -Map propertiesMap = ImmutableMap.builder() - .put("k1", "v1") - .build(); +Map filesetProperties = ImmutableMap.builder() + .put("k1", "v1") + .build(); filesetCatalog.createFileset( - NameIdentifier.of("test_schema", "example_fileset"), + NameIdentifier.of("s3_schema", "example_fileset"), "This is an example fileset", Fileset.Type.MANAGED, "s3a://bucket/root/schema/example_fileset", - propertiesMap -); + filesetProperties); ``` ```python -gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake") - -catalog: Catalog = gravitino_client.load_catalog(name="catalog") -catalog.as_fileset_catalog().create_fileset(ident=NameIdentifier.of("schema", "example_fileset"), - type=Fileset.Type.MANAGED, - comment="This is an example fileset", - storage_location="s3a://bucket/root/schema/example_fileset", - properties={"k1": "v1"}) +catalog: Catalog = gravitino_client.load_catalog(name="s3_catalog") +catalog.as_fileset_catalog().create_fileset( + ident=NameIdentifier.of("s3_schema", "example_fileset"), + type=Fileset.Type.MANAGED, + comment="This is an example fileset", + storage_location="s3a://bucket/root/schema/example_fileset", + properties={"k1": "v1"}) ``` -## Access a Fileset with S3 +The fileset is now addressable as +`gvfs://fileset/s3_catalog/s3_schema/example_fileset` from any GVFS client. -### Access the Fileset with the GVFS Java Client +## Access the Fileset -To access fileset with S3 using the GVFS Java client, based on the [basic GVFS configurations](./how-to-use-gvfs.md#configuration-1), you need to add the following configurations: +### Java client jars -| Configuration item | Description | Default value | Required | -|------------------------|-------------------------------|---------------|----------| -| `s3-endpoint` | The endpoint of the AWS S3. | (none) | Yes | -| `s3-access-key-id` | The access key of the AWS S3. | (none) | Yes | -| `s3-secret-access-key` | The secret key of the AWS S3. | (none) | Yes | +Every Java or Hadoop-based client needs `gravitino-filesystem-hadoop3-runtime`, which is published +on Maven Central, plus the Amazon S3 filesystem implementation. Only the latter differs by +environment: -:::note -- If the catalog has enabled [credential vending](security/credential-vending.md), the properties above can be omitted. More details can be found in [Fileset with credential vending](#fileset-with-credential-vending). -::: +| Environment | Jar providing the Amazon S3 filesystem | +|------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| No Hadoop installed | [`gravitino-aws-bundle`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aws-bundle), a fat jar bundling the Amazon S3 filesystem implementation and the AWS SDK | +| Hadoop already present | `hadoop-aws-${hadoop-version}.jar` and `aws-java-sdk-bundle-1.12.262.jar`, shipped with Hadoop under `${HADOOP_HOME}/share/hadoop/tools/lib` | -```java -Configuration conf = new Configuration(); -conf.set("fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs"); -conf.set("fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem"); -conf.set("fs.gravitino.server.uri", "http://localhost:8090"); -conf.set("fs.gravitino.client.metalake", "test_metalake"); -conf.set("s3-endpoint", "http://localhost:9000"); -conf.set("s3-access-key-id", "minio"); -conf.set("s3-secret-access-key", "minio123"); +The artifacts in full: -Path filesetPath = new Path("gvfs://fileset/fileset_catalog/fileset_schema/my_fileset/new_dir"); -FileSystem fs = filesetPath.getFileSystem(conf); -fs.mkdirs(filesetPath); -... -``` +- [`gravitino-aws-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aws-bundle): + a "fat" jar that includes the `gravitino-aws` functionality together with every dependency it needs, + such as `hadoop-aws` and the AWS SDK. Use it when the environment has no pre-existing Hadoop setup. +- [`gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-filesystem-hadoop3-runtime): + a "fat" jar that bundles the Gravitino virtual filesystem client and already includes the + `gravitino-aws` functionality. Java and Hadoop-based clients require it to access Gravitino + filesets. +- `hadoop-aws-${hadoop-version}.jar` and `aws-java-sdk-bundle-1.12.262.jar`: the standard Hadoop + dependencies for Amazon S3 access, shipped with Hadoop under + `${HADOOP_HOME}/share/hadoop/tools/lib`. Supply them yourself when running inside an existing + Hadoop environment. +- [`gravitino-aws-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aws): + a "thin" jar carrying only the AWS integration code. It is already contained in both jars above, + so it is not needed as a direct dependency unless you prefer to manage all Hadoop and AWS + dependencies yourself. -Similar to Spark configurations, you need to add S3 (bundle) jars to the classpath according to your environment. +```xml + + + org.apache.gravitino + gravitino-aws-bundle + ${GRAVITINO_VERSION} + + + org.apache.gravitino + gravitino-filesystem-hadoop3-runtime + ${GRAVITINO_VERSION} + +``` ```xml - - org.apache.hadoop - hadoop-common - ${HADOOP_VERSION} - - - - org.apache.hadoop - hadoop-aws - ${HADOOP_VERSION} - - - - org.apache.gravitino - gravitino-filesystem-hadoop3-runtime - ${GRAVITINO_VERSION} - + + + org.apache.hadoop + hadoop-common + ${HADOOP_VERSION} + + + org.apache.hadoop + hadoop-aws + ${HADOOP_VERSION} + + + org.apache.gravitino + gravitino-filesystem-hadoop3-runtime + ${GRAVITINO_VERSION} + ``` :::note -The `gravitino-aws` JAR is no longer required, as it is now included in the `gravitino-filesystem-hadoop3-runtime` JAR. +The thin `gravitino-aws` jar is not needed. Its functionality is already included in both +`gravitino-aws-bundle` and `gravitino-filesystem-hadoop3-runtime`. ::: -Or use the bundle jar with Hadoop environment if there is no Hadoop environment: +### GVFS Java client +On top of the [base GVFS configuration](./how-to-use-gvfs.md#configuration), set the Amazon S3 +properties from the table above. -```xml - - org.apache.gravitino - gravitino-aws-bundle - ${GRAVITINO_VERSION} - - - - org.apache.gravitino - gravitino-filesystem-hadoop3-runtime - ${GRAVITINO_VERSION} - -``` +```java +Configuration conf = new Configuration(); +conf.set("fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs"); +conf.set("fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem"); +conf.set("fs.gravitino.server.uri", "http://localhost:8090"); +conf.set("fs.gravitino.client.metalake", "metalake"); +conf.set("s3-endpoint", "http://s3.ap-northeast-1.amazonaws.com"); +conf.set("s3-access-key-id", "access_key"); +conf.set("s3-secret-access-key", "secret_key"); -### Access the Fileset with Spark +Path filesetPath = new Path("gvfs://fileset/s3_catalog/s3_schema/example_fileset/new_dir"); +FileSystem fs = filesetPath.getFileSystem(conf); +fs.mkdirs(filesetPath); +``` -The following Python code demonstrates how to use **PySpark 3.5.0 with Hadoop environment(Hadoop 3.3.4)** to access the fileset: +### Apache Spark -Before running the following code, you need to install required packages: +The example below uses PySpark 3.5.0 in an environment that already has Hadoop 3.3.4. ```bash pip install pyspark==3.5.0 pip install apache-gravitino==${GRAVITINO_VERSION} ``` -Then you can run the following code: ```python -from pyspark.sql import SparkSession import os +from pyspark.sql import SparkSession -gravitino_url = "http://localhost:8090" -metalake_name = "test" - -catalog_name = "your_s3_catalog" -schema_name = "your_s3_schema" -fileset_name = "your_s3_fileset" - -# JDK8 as follows, JDK17 will be slightly different, you need to add '--conf \"spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED\" --conf \"spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED\"' to the submit args. -os.environ["PYSPARK_SUBMIT_ARGS"] = "--jars /path/to/gravitino-filesystem-hadoop3-runtime-${gravitino-version}-SNAPSHOT.jar,/path/to/hadoop-aws-3.3.4.jar,/path/to/aws-java-sdk-bundle-1.12.262.jar --master local[1] pyspark-shell" -spark = SparkSession.builder - .appName("s3_fileset_test") +# On JDK 17, also add: +# --conf "spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" +# --conf "spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" +os.environ["PYSPARK_SUBMIT_ARGS"] = ( + "--jars /path/to/gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar," + "/path/to/hadoop-aws-3.3.4.jar," + "/path/to/aws-java-sdk-bundle-1.12.262.jar " + "--master local[1] pyspark-shell" +) + +spark = (SparkSession.builder + .appName("s3_fileset") .config("spark.hadoop.fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs") .config("spark.hadoop.fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem") .config("spark.hadoop.fs.gravitino.server.uri", "http://localhost:8090") - .config("spark.hadoop.fs.gravitino.client.metalake", "test") - .config("spark.hadoop.s3-access-key-id", os.environ["S3_ACCESS_KEY_ID"]) - .config("spark.hadoop.s3-secret-access-key", os.environ["S3_SECRET_ACCESS_KEY"]) + .config("spark.hadoop.fs.gravitino.client.metalake", "metalake") .config("spark.hadoop.s3-endpoint", "http://s3.ap-northeast-1.amazonaws.com") + .config("spark.hadoop.s3-access-key-id", "access_key") + .config("spark.hadoop.s3-secret-access-key", "secret_key") .config("spark.driver.memory", "2g") .config("spark.driver.port", "2048") - .getOrCreate() + .getOrCreate()) data = [("Alice", 25), ("Bob", 30), ("Cathy", 45)] -columns = ["Name", "Age"] -spark_df = spark.createDataFrame(data, schema=columns) -gvfs_path = f"gvfs://fileset/{catalog_name}/{schema_name}/{fileset_name}/people" - -spark_df.coalesce(1).write - .mode("overwrite") - .option("header", "true") - .csv(gvfs_path) -``` +spark_df = spark.createDataFrame(data, schema=["Name", "Age"]) +gvfs_path = "gvfs://fileset/s3_catalog/s3_schema/example_fileset/people" -If your Spark **without Hadoop environment**, you can use the following code snippet to access the fileset: - -```python -## Replace the following code snippet with the above code snippet with the same environment variables -os.environ["PYSPARK_SUBMIT_ARGS"] = "--jars /path/to/gravitino-aws-bundle-${gravitino-version}.jar,/path/to/gravitino-filesystem-hadoop3-runtime-${gravitino-version}-SNAPSHOT.jar --master local[1] pyspark-shell" +spark_df.coalesce(1).write.mode("overwrite").option("header", "true").csv(gvfs_path) ``` -- [`gravitino-aws-bundle-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aws-bundle): A "fat" JAR that includes `gravitino-aws` functionality and all necessary dependencies like `hadoop-aws` (3.3.1) and the `AWS SDK`. Use this if your Spark environment doesn't have a pre-existing Hadoop setup. -- [`gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-filesystem-hadoop3-runtime): A "fat" JAR that bundles Gravitino's virtual filesystem client and includes the functionality of `gravitino-aws`. It is required for accessing Gravitino filesets. -- `hadoop-aws-3.3.4.jar` and `aws-java-sdk-bundle-1.12.262.jar`: Standard Hadoop dependencies for S3 access. If you are running in an existing Hadoop environment, you need to provide these JARs. They are typically located in the `${HADOOP_HOME}/share/hadoop/tools/lib` directory. -- [`gravitino-aws-${gravitino-version}.jar`](https://mvnrepository.com/artifact/org.apache.gravitino/gravitino-aws): A "thin" JAR that only provides the AWS integration code. Its functionality is already included in the `gravitino-aws-bundle` and `gravitino-filesystem-hadoop3-runtime` JARs, so you do not need to add it as a direct dependency unless you want to manage all Hadoop and AWS dependencies manually. +If Spark runs without a Hadoop environment, only the jar list changes: -Please choose the correct jar according to your environment. +```python +os.environ["PYSPARK_SUBMIT_ARGS"] = ( + "--jars /path/to/gravitino-aws-bundle-${gravitino-version}.jar," + "/path/to/gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar " + "--master local[1] pyspark-shell" +) +``` :::note -In some Spark versions, a Hadoop environment is needed by the driver, adding the bundle jars with '--jars' may not work. If this is the case, you should add the jars to the spark CLASSPATH directly. +Some Spark versions need a Hadoop environment in the driver and do not pick up filesystem +implementations passed with `--jars`. If that happens, add the jars to the Spark classpath directly. ::: -### Access a Fileset Using the Hadoop Fs Command +### Hadoop fs command -The following are examples of how to use the `hadoop fs` command to access the fileset in Hadoop 3.1.3: - -1. Adding the following contents to the `${HADOOP_HOME}/etc/hadoop/core-site.xml` file: +1. Add the following to `${HADOOP_HOME}/etc/hadoop/core-site.xml`: ```xml - - fs.AbstractFileSystem.gvfs.impl - org.apache.gravitino.filesystem.hadoop.Gvfs - - - - fs.gvfs.impl - org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem - - - - fs.gravitino.server.uri - http://localhost:8090 - - - - fs.gravitino.client.metalake - test - - - - s3-endpoint - http://s3.ap-northeast-1.amazonaws.com - - - - s3-access-key-id - access-key - - - + + fs.AbstractFileSystem.gvfs.impl + org.apache.gravitino.filesystem.hadoop.Gvfs + + + fs.gvfs.impl + org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem + + + fs.gravitino.server.uri + http://localhost:8090 + + + fs.gravitino.client.metalake + metalake + + + s3-endpoint + http://s3.ap-northeast-1.amazonaws.com + + + s3-access-key-id + access_key + + s3-secret-access-key - secret-key - + secret_key + ``` -2. Add the necessary jars to the Hadoop classpath. +2. Add these jars to the Hadoop classpath: -For S3, you need to add `gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar` and `hadoop-aws-${hadoop-version}.jar` located at `${HADOOP_HOME}/share/hadoop/tools/lib/` to Hadoop classpath. + - `gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar`, from Maven Central. + - `hadoop-aws-${hadoop-version}.jar` and `aws-java-sdk-bundle-1.12.262.jar`, shipped with Hadoop under `${HADOOP_HOME}/share/hadoop/tools/lib`. -3. Run the following command to access the fileset: +3. Access the fileset: ```shell -./${HADOOP_HOME}/bin/hadoop dfs -ls gvfs://fileset/s3_catalog/s3_schema/s3_fileset -./${HADOOP_HOME}/bin/hadoop dfs -put /path/to/local/file gvfs://fileset/s3_catalog/s3_schema/s3_fileset +${HADOOP_HOME}/bin/hadoop fs -ls gvfs://fileset/s3_catalog/s3_schema/example_fileset +${HADOOP_HOME}/bin/hadoop fs -put /path/to/local/file gvfs://fileset/s3_catalog/s3_schema/example_fileset ``` -### Access the Fileset with the GVFS Python Client - -To access fileset with S3 using the GVFS Python client, apart from [basic GVFS configurations](./how-to-use-gvfs.md#configuration-1), you need to add the following configurations: - -| Configuration item | Description | Default value | Required | -|------------------------|----------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------| -| `s3_endpoint` | The endpoint of the AWS S3. This configuration is optional for S3 service, but required for other S3-compatible storage services like MinIO. | (none) | No | -| `s3_access_key_id` | The access key of the AWS S3. | (none) | Yes | -| `s3_secret_access_key` | The secret key of the AWS S3. | (none) | Yes | - -:::note -- `s3_endpoint` is an optional configuration for GVFS **Python** client but a required configuration for GVFS **Java** client to access Hadoop with AWS S3, and it is required for other S3-compatible storage services like MinIO. -- If the catalog has enabled [credential vending](security/credential-vending.md), the properties above can be omitted. -::: - -Please install the `gravitino` package before running the following code: +### GVFS Python client ```bash pip install apache-gravitino==${GRAVITINO_VERSION} ``` +On top of the [base GVFS configuration](./how-to-use-gvfs.md#configuration-1), pass the Amazon S3 +properties in `options`, spelled with underscores. + ```python from gravitino import gvfs + options = { "cache_size": 20, "cache_expired_time": 3600, "auth_type": "simple", - "s3_endpoint": "http://localhost:9000", - "s3_access_key_id": "minio", - "s3_secret_access_key": "minio123" + "s3_endpoint": "http://s3.ap-northeast-1.amazonaws.com", + "s3_access_key_id": "access_key", + "s3_secret_access_key": "secret_key", } -fs = gvfs.GravitinoVirtualFileSystem(server_uri="http://localhost:8090", metalake_name="test_metalake", options=options) -fs.ls("gvfs://fileset/{catalog_name}/{schema_name}/{fileset_name}/") ") + +fs = gvfs.GravitinoVirtualFileSystem(server_uri="http://localhost:8090", + metalake_name="metalake", + options=options) +fs.ls("gvfs://fileset/s3_catalog/s3_schema/example_fileset/") ``` -### Access the Fileset with Pandas +### pandas -The following are examples of how to use the pandas library to access the S3 fileset +pandas reaches the same paths through `storage_options`. Use the `fs` instance from the preceding +GVFS example to discover the generated Spark part file. ```python import pandas as pd storage_options = { - "server_uri": "http://localhost:8090", - "metalake_name": "test", + "server_uri": "http://localhost:8090", + "metalake_name": "metalake", "options": { + "s3_endpoint": "http://s3.ap-northeast-1.amazonaws.com", "s3_access_key_id": "access_key", "s3_secret_access_key": "secret_key", - "s3_endpoint": "http://s3.ap-northeast-1.amazonaws.com" } } -ds = pd.read_csv(f"gvfs://fileset/${catalog_name}/${schema_name}/${fileset_name}/people/part-00000-51d366e2-d5eb-448d-9109-32a96c8a14dc-c000.csv", - storage_options=storage_options) + +csv_path = next( + f"gvfs://{path}" + for path in fs.ls( + "gvfs://fileset/s3_catalog/s3_schema/example_fileset/people", + detail=False, + ) + if ( + path.rsplit("/", 1)[-1].startswith("part-") + and path.endswith(".csv") + ) +) +ds = pd.read_csv(csv_path, storage_options=storage_options) ds.head() ``` -For more use cases, refer to the [Gravitino Virtual File System](./how-to-use-gvfs.md) document. +For further use cases, see [Gravitino Virtual File System](./how-to-use-gvfs.md). -## Fileset with Credential Vending +## Credential Vending -Gravitino supports credential vending for S3 fileset. If the catalog has been [configured with credential](./security/credential-vending.md), you can access S3 fileset without providing authentication information like `s3-access-key-id` and `s3-secret-access-key` in the properties. +With credential vending the catalog holds the Amazon S3 credentials and the Gravitino server hands +out a credential per request, so clients never hold cloud keys of their own. See +[Credential Vending](./security/credential-vending.md) for the general mechanism and +[S3 credentials](./security/credential-vending.md#s3) for the properties +each provider takes. -### Create an S3 Fileset Catalog with Credential Vending +The supported providers are `s3-token`, which vends a short-lived STS token; +`s3-secret-key`, which vends the static access key configured on the catalog; and `aws-irsa`, which +vends credentials from an IAM role for service accounts and currently reads the web identity token +from a file. The example below uses `s3-token`. -Apart from configuration method in [create-s3-fileset-catalog](#s3-fileset-catalog-configuration), -properties needed by [s3-credential](./security/credential-vending.md#s3-credentials) -should also be set to enable credential vending for S3 fileset. Take `s3-token` credential provider for example: +### Configure the catalog, schema, and fileset ```shell curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ -H "Content-Type: application/json" -d '{ - "name": "s3-catalog-with-token", + "name": "s3_catalog_with_vending", "type": "FILESET", - "comment": "This is a S3 fileset catalog", + "comment": "A fileset catalog backed by Amazon S3 with credential vending", "properties": { "location": "s3a://bucket/root", + "s3-endpoint": "http://s3.ap-northeast-1.amazonaws.com", "s3-access-key-id": "access_key", "s3-secret-access-key": "secret_key", - "s3-endpoint": "http://s3.ap-northeast-1.amazonaws.com", "credential-providers": "s3-token", - "s3-region":"ap-northeast-1", - "s3-role-arn":"The ARN of the role to access the S3 data" + "s3-region": "ap-northeast-1", + "s3-role-arn": "arn:aws:iam::123456789012:role/gravitino-fileset" } }' http://localhost:8090/api/metalakes/metalake/catalogs ``` -### Access an S3 Fileset with Credential Vending +Create the schema and fileset in the credential-vending catalog: + +```shell +curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ +-H "Content-Type: application/json" -d '{ + "name": "s3_schema", + "comment": "A schema in the Amazon S3 credential-vending catalog", + "properties": { + "location": "s3a://bucket/root/schema" + } +}' http://localhost:8090/api/metalakes/metalake/catalogs/s3_catalog_with_vending/schemas + +curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \ +-H "Content-Type: application/json" -d '{ + "name": "example_fileset", + "comment": "This is an example fileset", + "type": "MANAGED", + "storageLocation": "s3a://bucket/root/schema/example_fileset", + "properties": {} +}' http://localhost:8090/api/metalakes/metalake/catalogs/s3_catalog_with_vending/schemas/s3_schema/filesets +``` + +The `s3-token` provider needs two more catalog properties. + +| Property Name | Description | +|---------------|----------------------------------------------------| +| `s3-region` | Region of the bucket, for example `ap-northeast-1` | +| `s3-role-arn` | ARN of the role that grants access to the data | -When the catalog is configured with credentials and client-side credential vending is enabled, -you can access S3 filesets directly using the GVFS Java/Python client or Spark without providing authentication details. +### Access without local credentials -GVFS Java client: +Enable vending on the client and drop the credential properties. ```java Configuration conf = new Configuration(); @@ -540,29 +563,34 @@ conf.setBoolean("fs.gravitino.enableCredentialVending", true); conf.set("fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs"); conf.set("fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem"); conf.set("fs.gravitino.server.uri", "http://localhost:8090"); -conf.set("fs.gravitino.client.metalake", "test_metalake"); -// No need to set s3-access-key-id and s3-secret-access-key -Path filesetPath = new Path("gvfs://fileset/test_catalog/test_schema/test_fileset/new_dir"); +conf.set("fs.gravitino.client.metalake", "metalake"); +// No need to set s3-access-key-id or s3-secret-access-key + +Path filesetPath = new Path( + "gvfs://fileset/s3_catalog_with_vending/s3_schema/example_fileset/new_dir"); FileSystem fs = filesetPath.getFileSystem(conf); fs.mkdirs(filesetPath); -... ``` -Spark: - ```python -spark = SparkSession.builder - .appName("s3_fileset_test") +spark = (SparkSession.builder + .appName("s3_fileset") .config("spark.hadoop.fs.gravitino.enableCredentialVending", "true") .config("spark.hadoop.fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs") .config("spark.hadoop.fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem") .config("spark.hadoop.fs.gravitino.server.uri", "http://localhost:8090") - .config("spark.hadoop.fs.gravitino.client.metalake", "test") - # No need to set s3-access-key-id and s3-secret-access-key - .config("spark.driver.memory", "2g") - .config("spark.driver.port", "2048") - .getOrCreate() + .config("spark.hadoop.fs.gravitino.client.metalake", "metalake") + # No need to set s3-access-key-id or s3-secret-access-key + .getOrCreate()) ``` -Python client and Hadoop command are similar to the above examples. - +```python +options = { + "auth_type": "simple", + "enable_credential_vending": True, + # No need to set s3-access-key-id or s3-secret-access-key +} +fs = gvfs.GravitinoVirtualFileSystem(server_uri="http://localhost:8090", + metalake_name="metalake", + options=options) +``` diff --git a/docs/fileset-catalog.md b/docs/fileset-catalog.md index c096e71d0a8..2bcbdf1a522 100644 --- a/docs/fileset-catalog.md +++ b/docs/fileset-catalog.md @@ -8,15 +8,15 @@ license: "This software is licensed under the Apache License version 2." ## Introduction -Fileset catalog is a fileset catalog that using Hadoop Compatible File System (HCFS) to manage -the storage location of the fileset. It supports the local filesystem and HDFS. -Gravitino supports [S3](fileset-catalog-with-s3.md), [GCS](fileset-catalog-with-gcs.md), -[OSS](fileset-catalog-with-oss.md) and [Azure Blob Storage](fileset-catalog-with-adls.md) through Fileset catalog. -Gravitino also supports [Tencent Cloud COS](fileset-catalog-with-cos.md). +The fileset catalog manages the storage location of a fileset through a Hadoop Compatible File +System (HCFS). It supports the local filesystem and HDFS out of the box, and Amazon S3, Google Cloud +Storage, Azure Data Lake Storage, Alibaba Cloud OSS and Tencent Cloud COS once the matching bundle +jar is on the classpath. -The rest of this document will use HDFS or local file as an example to illustrate how to use the Fileset catalog. -For S3, GCS, OSS, Azure Blob Storage and COS, the configuration is similar to HDFS, -refer to the corresponding document for more details. +This page is the shared reference: the properties every backend accepts, how they are inherited from +catalog to schema to fileset, and how to plug in a custom filesystem. It uses HDFS and the local +filesystem in its examples. For a runnable end-to-end example on a cloud backend, follow the page for +that backend listed under [Fileset Catalog with Cloud Storage](#fileset-catalog-with-cloud-storage). Note that Gravitino uses Hadoop 3 dependencies to build Fileset catalog. Theoretically, it should be compatible with both Hadoop 2.x and 3.x, since Gravitino doesn't leverage any new features in @@ -26,7 +26,7 @@ Hadoop 3. If there's any compatibility issue, create an [issue](https://github.c ### Catalog Properties -Besides the [common catalog properties](./gravitino-server-config.md#catalog-properties-configuration), +Besides the [common catalog properties](./gravitino-server-config.md#catalog-properties), the Fileset catalog has the following properties: | Property Name | Description | Default Value | Required | @@ -75,15 +75,23 @@ The Gravitino Fileset extends the following properties in the `xxx-site.xml`: ### Fileset Catalog with Cloud Storage -In the current implementation, the fileset uses the HDFS protocol to access its location. If users use S3, GCS, OSS, -Azure Blob Storage or Tencent Cloud COS, they can also configure the `config.resources` to specify custom configuration -files. +For Java and Hadoop-based access, Fileset uses the Hadoop Compatible File System (HCFS) interface. +Each cloud backend provides its own Hadoop `FileSystem` implementation, such as S3A for Amazon S3. +Put the matching bundle jar on the classpath and set the credential properties for that backend. +Python clients use fsspec-based implementations and do not require these jars. Each backend has its +own page with a runnable end-to-end example. -- For S3, refer to [Fileset-catalog-with-s3](./fileset-catalog-with-s3.md) for more details. -- For GCS, refer to [Fileset-catalog-with-gcs](./fileset-catalog-with-gcs.md) for more details. -- For OSS, refer to [Fileset-catalog-with-oss](./fileset-catalog-with-oss.md) for more details. -- For Azure Blob Storage, refer to [Fileset-catalog-with-adls](./fileset-catalog-with-adls.md) for more details. -- For Tencent Cloud COS, refer to [Fileset-catalog-with-cos](./fileset-catalog-with-cos.md) for more details. +| Storage backend | Bundle jar | Location scheme | Backend properties | +|-----------------------------------------------------------|----------------------------|-----------------|----------------------------------------------------------------------------| +| [Amazon S3](./fileset-catalog-with-s3.md) | `gravitino-aws-bundle` | `s3a://` | `s3-endpoint`, `s3-access-key-id`, `s3-secret-access-key` | +| [Google Cloud Storage](./fileset-catalog-with-gcs.md) | `gravitino-gcp-bundle` | `gs://` | `gcs-service-account-file` | +| [Azure Data Lake Storage](./fileset-catalog-with-adls.md) | `gravitino-azure-bundle` | `abfss://` | `azure-storage-account-name`, `azure-storage-account-key` | +| [Alibaba Cloud OSS](./fileset-catalog-with-oss.md) | `gravitino-aliyun-bundle` | `oss://` | `oss-endpoint`, `oss-access-key-id`, `oss-secret-access-key` | +| [Tencent Cloud COS](./fileset-catalog-with-cos.md) | `gravitino-tencent-bundle` | `cosn://` | `cos-region`, `cos-access-key-id`, `cos-secret-access-key`, `cos-endpoint` | + +A catalog may hold locations in more than one backend at the same time, as long as every bundle jar +involved is on the classpath. Cloud backends also accept `config.resources` to pass custom +configuration files to the underlying filesystem client. ### Implement a Custom HCFS File System Fileset @@ -134,7 +142,7 @@ value, and the priority mechanism is the same as authentication. ### Catalog Operations -Refer to [Catalog operations](./manage-fileset-metadata-using-gravitino.md#catalog-operations) for more details. +Refer to [Catalog operations](./manage-catalogs-and-schemas.md#catalog-operations) for more details. ## Schema @@ -160,7 +168,7 @@ properties: ### Schema Operations -Refer to [Schema operation](./manage-fileset-metadata-using-gravitino.md#schema-operations) for more details. +Refer to [Schema operations](./manage-catalogs-and-schemas.md#schema-operations) for more details. :::note During schema creation or deletion, Gravitino automatically creates or removes the corresponding filesystem directories @@ -168,7 +176,7 @@ for the schema locations. This behavior is skipped in either of these cases: 1. When the catalog property `disable-filesystem-ops` is set to `true` -2. When the location contains [placeholders](./manage-fileset-metadata-using-gravitino.md#placeholder) +2. When the location contains [placeholders](./filesets.md#storage-locations) ::: ## Fileset diff --git a/docs/hive-catalog-with-cloud-storage.md b/docs/hive-catalog-with-cloud-storage.md index 03f0f343bed..96b2b3ac3fa 100644 --- a/docs/hive-catalog-with-cloud-storage.md +++ b/docs/hive-catalog-with-cloud-storage.md @@ -100,7 +100,7 @@ Once all configurations have been correctly set, restart the Hive cluster to app ## Create Tables or Databases with S3 Storage Using Gravitino -Assuming you have already set up a Hive catalog with Gravitino, you can proceed to create tables or databases using S3 storage. For more information on catalog operations, refer to [Catalog operations](./manage-fileset-metadata-using-gravitino.md#catalog-operations) +Assuming you have already set up a Hive catalog with Gravitino, you can proceed to create tables or databases using S3 storage. For more information on catalog operations, refer to [Catalog operations](./manage-catalogs-and-schemas.md#catalog-operations) ### Example: Creating a Database with S3 Storage diff --git a/docs/how-to-use-gvfs.md b/docs/how-to-use-gvfs.md index 0f7e030a68c..257015c72c3 100644 --- a/docs/how-to-use-gvfs.md +++ b/docs/how-to-use-gvfs.md @@ -78,7 +78,7 @@ To configure the Gravitino client, use properties prefixed with `fs.gravitino.cl :::note When users work with a multi-cluster fileset catalog, they can configure separate sets of properties for the base paths -of the different clusters. [Manage filesets across multiple clusters](./manage-fileset-metadata-using-gravitino.md#manage-filesets-across-multiple-clusters) +of the different clusters, using the `fs.path.config.` properties described above. For example, a complex catalog structure might look like this: @@ -112,11 +112,14 @@ The plain `fs.path.config.` entry specifies the base path of the filesyste **Note:** Invalid configuration properties will result in exceptions. Please see [Gravitino Java client configurations](./how-to-use-gravitino-client.md#java-client-configuration) for more support client configuration. -Apart from the above properties, to access fileset like S3, GCS, OSS and custom fileset, extra properties are needed; see -[S3 GVFS Java client configurations](./fileset-catalog-with-s3.md#access-the-fileset-with-the-gvfs-java-client), -[GCS GVFS Java client configurations](./fileset-catalog-with-gcs.md#access-the-fileset-with-the-gvfs-java-client), -[OSS GVFS Java client configurations](./fileset-catalog-with-oss.md#access-the-fileset-with-the-gvfs-java-client) -and [Azure Blob Storage GVFS Java client configurations](./fileset-catalog-with-adls.md#access-the-fileset-with-the-gvfs-java-client) for more details. +A fileset backed by cloud storage needs the credential properties of that backend on top of the +properties above, and the matching bundle jar on the classpath. See +[Amazon S3](./fileset-catalog-with-s3.md#amazon-s3-properties), +[Google Cloud Storage](./fileset-catalog-with-gcs.md#google-cloud-storage-properties), +[Azure Data Lake Storage](./fileset-catalog-with-adls.md#azure-data-lake-storage-properties), +[Alibaba Cloud OSS](./fileset-catalog-with-oss.md#alibaba-cloud-oss-properties) and +[Tencent Cloud COS](./fileset-catalog-with-cos.md#tencent-cloud-cos-properties) for the property +names and a runnable example for each backend. #### Custom Fileset @@ -413,8 +416,8 @@ to recompile the native libraries like `libhdfs` and others, and completely repl | `cache_size` | The cache capacity of the Gravitino Virtual File System. | `20` | No | | `cache_expired_time` | The value of time that the cache expires after accessing in the Gravitino Virtual File System. The value is in `seconds`. | `3600` | No | | `auth_type` | The auth type the Gravitino client uses with the Gravitino Virtual File System. Supports `simple`, `basic`, and `oauth2`. | `simple` | No | -| `basic_username` | The username for the Gravitino client when using `basic` auth type with the local user store. | (none) | Yes if you use `basic` auth type | -| `basic_password` | The password for the Gravitino client when using `basic` auth type with the local user store. | (none) | Yes if you use `basic` auth type | +| `basic_username` | The username for the Gravitino client when using `basic` auth type with the local user store. | (none) | Yes if you use `basic` auth type | +| `basic_password` | The password for the Gravitino client when using `basic` auth type with the local user store. | (none) | Yes if you use `basic` auth type | | `oauth2_server_uri` | The auth server URI for the Gravitino client when using `oauth2` auth type. | (none) | Yes if you use `oauth2` auth type | | `oauth2_credential` | The auth credential for the Gravitino client when using `oauth2` auth type. | (none) | Yes if you use `oauth2` auth type | | `oauth2_path` | The auth server path for the Gravitino client when using `oauth2` auth type. Please remove the first slash `/` from the path, for example `oauth/token`. | (none) | Yes if you use `oauth2` auth type | @@ -439,7 +442,7 @@ To configure the Gravitino client, use properties prefixed with `gvfs_gravitino_ :::note When users work with a multi-cluster fileset catalog, they can configure separate sets of properties for the base paths -of the different clusters. [Manage filesets across multiple clusters](./manage-fileset-metadata-using-gravitino.md#manage-filesets-across-multiple-clusters) +of the different clusters, using the `fs_path_config_` properties described above. For example, a complex catalog structure might look like this: @@ -471,12 +474,16 @@ options = { The plain `fs_path_config_` entry specifies the base path of the filesystem. Any additional key under the same prefix (`fs_path_config__`) is treated as a location-scoped configuration (for example, `config.resource` for HDFS) and is forwarded directly to the underlying filesystem client. ::: -#### Configurations for S3, GCS, OSS and Azure Blob Storage Fileset +#### Configurations for Cloud Storage Filesets -Please see the cloud-storage-specific configurations [GCS GVFS Python client configurations](./fileset-catalog-with-gcs.md#access-the-fileset-with-the-gvfs-python-client), -[S3 GVFS Python client configurations](./fileset-catalog-with-s3.md#access-the-fileset-with-the-gvfs-python-client), -[OSS GVFS Python client configurations](./fileset-catalog-with-oss.md#access-the-fileset-with-the-gvfs-python-client) -and [Azure Blob Storage GVFS Python client configurations](./fileset-catalog-with-adls.md#access-the-fileset-with-the-gvfs-python-client) for more details. +A fileset backed by cloud storage needs the credential properties of that backend, spelled with +underscores rather than hyphens. See +[Amazon S3](./fileset-catalog-with-s3.md#amazon-s3-properties), +[Google Cloud Storage](./fileset-catalog-with-gcs.md#google-cloud-storage-properties), +[Azure Data Lake Storage](./fileset-catalog-with-adls.md#azure-data-lake-storage-properties), +[Alibaba Cloud OSS](./fileset-catalog-with-oss.md#alibaba-cloud-oss-properties) and +[Tencent Cloud COS](./fileset-catalog-with-cos.md#tencent-cloud-cos-properties) for the property +names and a runnable example for each backend. :::note Gravitino python client does not support [customized file systems](fileset-catalog.md#implement-a-custom-hcfs-file-system-fileset) defined by users due to the limit of `fsspec` library.