docs: warn that connector configs are stored in plaintext, and document secrets provider behaviour

docs/resources/sink.md

@@ -25,7 +25,7 @@ Manages Pulsar IO sinks through the Functions Worker API.
 ### Optional
 
 - `classname` (String) The sink's class name if archive is file-url-path (file://)
-- `configs` (String) User defined configs key/values (JSON string)
+- `configs` (String) User defined configs key/values (JSON string). Values here are stored and returned in plaintext, including in the function metadata topic, `pulsar-admin sinks get` output and Terraform state, so use `secrets` for API keys, passwords and service-account keys rather than putting them here.
 - `cpu` (Number) The CPU that needs to be allocated per sink instance (applicable only to Docker runtime)
 - `custom_runtime_options` (String) A string that encodes options to customize the runtime
 - `custom_schema_inputs` (Map of String) The map of input topics to Schema types or class names (as a JSON string)
@@ -41,7 +41,7 @@ Manages Pulsar IO sinks through the Functions Worker API.
 - `ram_mb` (Number) The RAM that need to be allocated per sink instance (applicable only to the process and Docker runtimes)
 - `retain_key_ordering` (Boolean) Sink consumes and processes messages in key order
 - `retain_ordering` (Boolean) Sink consumes and sinks messages in order
-- `secrets` (String) The map of secretName to an object that encapsulates how the secret is fetched by the underlying secrets provider
+- `secrets` (String) The map of secretName to an object that encapsulates how the secret is fetched by the underlying secrets provider. Each entry maps the name the connector reads to a `{"path": ..., "key": ...}` reference, so only the reference is stored in connector config, function metadata and Terraform state - never the value. Note that these references are resolved only by a runtime whose secrets provider understands them, such as the Kubernetes runtime's KubernetesSecretsProviderConfigurator. Under the default ClearTextSecretsProvider used by the process and standalone runtimes, apply succeeds but the connector receives no value for the secret at runtime.
 - `sink_type` (String) The sinks's connector provider
 - `subscription_name` (String) Pulsar source subscription name if user wants a specific subscription-name for input-topic consumer
 - `subscription_position` (String) Pulsar source subscription position if user wants to consume messages from the specified location (Latest, Earliest). Default to Earliest.

docs/resources/source.md

@@ -26,7 +26,7 @@ Manages Pulsar IO sources through the Functions Worker API.
 - `batch_builder` (String) BatchBuilder provides two types of batch construction methods, DEFAULT and KEY_BASED.
 - `classname` (String) The source's class name if archive is file-url-path (file://)
 - `compression_type` (String) Set the compression type for the producer. By default, message payloads are not compressed. Supported compression types are: LZ4, ZLIB, ZSTD, SNAPPY and NONE
-- `configs` (String) User defined configs key/values (JSON string)
+- `configs` (String) User defined configs key/values (JSON string). Values here are stored and returned in plaintext, including in the function metadata topic, `pulsar-admin sources get` output and Terraform state, so use `secrets` for API keys, passwords and service-account keys rather than putting them here.
 - `consumer_crypto_failure_action` (String) The desired action if consumer fail to decrypt data, one of FAIL, DISCARD, CONSUME
 - `cpu` (Number) The CPU that needs to be allocated per source instance (applicable only to Docker runtime)
 - `crypto_key_reader_classname` (String) The classname for the crypto key reader that can be used to access the keys in the keystore
@@ -43,7 +43,7 @@ Manages Pulsar IO sources through the Functions Worker API.
 - `ram_mb` (Number) The RAM that need to be allocated per source instance (applicable only to the process and Docker runtimes)
 - `runtime_flags` (String) User defined configs key/values (JSON string)
 - `schema_type` (String) The schema type (either a builtin schema like 'avro', 'json', etc.. or custom Schema class name to be used to encode messages emitted from the source
-- `secrets` (String) The map of secretName to an object that encapsulates how the secret is fetched by the underlying secrets provider
+- `secrets` (String) The map of secretName to an object that encapsulates how the secret is fetched by the underlying secrets provider. Each entry maps the name the connector reads to a `{"path": ..., "key": ...}` reference, so only the reference is stored in connector config, function metadata and Terraform state - never the value. Note that these references are resolved only by a runtime whose secrets provider understands them, such as the Kubernetes runtime's KubernetesSecretsProviderConfigurator. Under the default ClearTextSecretsProvider used by the process and standalone runtimes, apply succeeds but the connector receives no value for the secret at runtime.
 - `use_thread_local_producers` (Boolean) Whether to use thread local producers
 
 ### Read-Only

examples/sinks/main.tf

@@ -39,6 +39,30 @@ resource "pulsar_sink" "example" {
   cleanup_subscription = false
   auto_ack             = true
 
+  # Values in `configs` are stored and returned in plaintext, including in the
+  # function metadata topic, `pulsar-admin sinks get` output and Terraform
+  # state. Keep credentials out of it and reference them through the worker's
+  # secrets provider instead:
+  #
+  #   configs = jsonencode({
+  #     jdbcUrl   = "jdbc:postgresql://localhost:5432/pulsar"
+  #     tableName = "events"
+  #     userName  = "postgres"
+  #   })
+  #
+  #   secrets = jsonencode({
+  #     password = {
+  #       path = "postgres-credentials"
+  #       key  = "password"
+  #     }
+  #   })
+  #
+  # Note that `{path, key}` references are resolved only by a runtime whose
+  # secrets provider understands them, such as the Kubernetes runtime's
+  # KubernetesSecretsProviderConfigurator. Under the default
+  # ClearTextSecretsProvider - which includes the standalone cluster started by
+  # `make run-pulsar-in-docker` - apply succeeds but the sink receives no value
+  # for the secret at runtime.
   configs = jsonencode({
     jdbcUrl   = "jdbc:postgresql://localhost:5432/pulsar"
     tableName = "events"

pulsar/resource_pulsar_sink.go

@@ -93,7 +93,7 @@ func init() {
 		resourceSinkCPUKey:                          "The CPU that needs to be allocated per sink instance (applicable only to Docker runtime)",
 		resourceSinkRAMKey:                          "The RAM that need to be allocated per sink instance (applicable only to the process and Docker runtimes)",
 		resourceSinkDiskKey:                         "The disk that need to be allocated per sink instance (applicable only to Docker runtime)",
-		resourceSinkConfigsKey:                      "User defined configs key/values (JSON string)",
+		resourceSinkConfigsKey:                      "User defined configs key/values (JSON string). Values here are stored and returned in plaintext, including in the function metadata topic, `pulsar-admin sinks get` output and Terraform state, so use `secrets` for API keys, passwords and service-account keys rather than putting them here.",
 		resourceSinkAutoACKKey:                      "Whether or not the framework will automatically acknowledge messages",
 		resourceSinkTimeoutKey:                      "The message timeout in milliseconds",
 		resourceSinkCustomRuntimeOptionsKey:         "A string that encodes options to customize the runtime",
@@ -102,7 +102,7 @@ func init() {
 		resourceSinkNegativeCountRedeliveryDelayKey: "The negative ack message redelivery delay in milliseconds",
 		resourceSinkRetainKeyOrderingKey:            "Sink consumes and processes messages in key order",
 		resourceSinkSinkTypeKey:                     "The sinks's connector provider",
-		resourceSinkSecretsKey:                      "The map of secretName to an object that encapsulates how the secret is fetched by the underlying secrets provider",
+		resourceSinkSecretsKey:                      "The map of secretName to an object that encapsulates how the secret is fetched by the underlying secrets provider. Each entry maps the name the connector reads to a `{\"path\": ..., \"key\": ...}` reference, so only the reference is stored in connector config, function metadata and Terraform state - never the value. Note that these references are resolved only by a runtime whose secrets provider understands them, such as the Kubernetes runtime's KubernetesSecretsProviderConfigurator. Under the default ClearTextSecretsProvider used by the process and standalone runtimes, apply succeeds but the connector receives no value for the secret at runtime.",
 	}
 }
 

pulsar/resource_pulsar_source.go

@@ -82,11 +82,11 @@ func init() {
 		resourceSourceCPUKey:                            "The CPU that needs to be allocated per source instance (applicable only to Docker runtime)",
 		resourceSourceRAMKey:                            "The RAM that need to be allocated per source instance (applicable only to the process and Docker runtimes)",
 		resourceSourceDiskKey:                           "The disk that need to be allocated per source instance (applicable only to Docker runtime)",
-		resourceSourceConfigsKey:                        "User defined configs key/values (JSON string)",
+		resourceSourceConfigsKey:                        "User defined configs key/values (JSON string). Values here are stored and returned in plaintext, including in the function metadata topic, `pulsar-admin sources get` output and Terraform state, so use `secrets` for API keys, passwords and service-account keys rather than putting them here.",
 		resourceSourceRuntimeFlagsKey:                   "User defined configs key/values (JSON string)",
 		resourceSourceCustomRuntimeOptionsKey:           "A string that encodes options to customize the runtime, see docs for configured runtime for details",
 		resourceSourceSchemaTypeKey:                     "The schema type (either a builtin schema like 'avro', 'json', etc.. or custom Schema class name to be used to encode messages emitted from the source",
-		resourceSourceSecretsKey:                        "The map of secretName to an object that encapsulates how the secret is fetched by the underlying secrets provider",
+		resourceSourceSecretsKey:                        "The map of secretName to an object that encapsulates how the secret is fetched by the underlying secrets provider. Each entry maps the name the connector reads to a `{\"path\": ..., \"key\": ...}` reference, so only the reference is stored in connector config, function metadata and Terraform state - never the value. Note that these references are resolved only by a runtime whose secrets provider understands them, such as the Kubernetes runtime's KubernetesSecretsProviderConfigurator. Under the default ClearTextSecretsProvider used by the process and standalone runtimes, apply succeeds but the connector receives no value for the secret at runtime.",
 		resourceSourcePCMaxPendingMsgKey:                "The maximum size of a queue holding pending messages",
 		resourceSourcePCMaxPendingMsgAcrossPartitionKey: "The maximum number of pending messages across partitions",
 		resourceSourcePCUseThreadLocalProducersKey:      "Whether to use thread local producers",