feat(source-ashby): add application_history stream #84392
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,15 @@ | ||
| version: 4.3.0 | ||
| type: DeclarativeSource | ||
| api_budget: | ||
| type: HTTPAPIBudget | ||
| policies: | ||
| - type: MovingWindowCallRatePolicy | ||
| rates: | ||
| - limit: 100 | ||
| interval: PT1M | ||
| matchers: | ||
| - type: HttpRequestRegexMatcher | ||
| url_path_pattern: /application\.listHistory | ||
| check: | ||
| type: CheckStream | ||
| stream_names: | ||
|
|
@@ -1359,6 +1369,7 @@ streams: | |
| fields: | ||
| - path: | ||
| - application_id | ||
| type: AddedFieldDefinition | ||
| value: "{{ stream_partition.application_id }}" | ||
| schema_loader: | ||
| type: InlineSchemaLoader | ||
|
|
@@ -1391,6 +1402,204 @@ streams: | |
| type: | ||
| - "null" | ||
| - string | ||
| - type: DeclarativeStream | ||
| name: application_history | ||
| primary_key: | ||
| - id | ||
| retriever: | ||
| type: SimpleRetriever | ||
| requester: | ||
| type: HttpRequester | ||
| url_base: https://api.ashbyhq.com | ||
| authenticator: | ||
| type: BasicHttpAuthenticator | ||
| username: "{{ config['api_key'] }}" | ||
| password: "{{ config['api_key'] }}" | ||
| path: /application.listHistory | ||
| http_method: POST | ||
| request_body_json: | ||
| applicationId: "{{ stream_partition.application_id }}" | ||
| error_handler: | ||
| type: DefaultErrorHandler | ||
| response_filters: | ||
| - type: HttpResponseFilter | ||
| action: RATE_LIMITED | ||
| http_codes: | ||
| - 429 | ||
| - type: HttpResponseFilter | ||
| action: RETRY | ||
| http_codes: | ||
| - 500 | ||
| - 502 | ||
| - 503 | ||
| - 504 | ||
| - type: HttpResponseFilter | ||
| action: IGNORE | ||
| predicate: >- | ||
| {{ response.get('success', true) == false and | ||
| response.get('errorInfo', {}).get('code') == 'application_not_found' }} | ||
| error_message: >- | ||
| application.listHistory skipped this application's history | ||
| (Ashby request ID: | ||
| {{ response.get('errorInfo', {}).get('requestId', 'unavailable') }}). | ||
| - type: HttpResponseFilter | ||
| action: FAIL | ||
| predicate: "{{ response.get('success', true) == false }}" | ||
| error_message: >- | ||
| application.listHistory failed with code | ||
| {{ response.get('errorInfo', {}).get('code', 'unknown') }}: | ||
| {{ response.get('errorInfo', {}).get('message') or | ||
| (response.get('errors') and response.get('errors')[0].get('message')) or | ||
| 'Unknown error' }}. | ||
| record_selector: | ||
| type: RecordSelector | ||
| extractor: | ||
| type: DpathExtractor | ||
| field_path: | ||
| - results | ||
| paginator: | ||
| type: DefaultPaginator | ||
| page_token_option: | ||
| type: RequestOption | ||
| inject_into: body_json | ||
| field_name: cursor | ||
| page_size_option: | ||
| type: RequestOption | ||
| inject_into: body_json | ||
| field_name: limit | ||
| pagination_strategy: | ||
| type: CursorPagination | ||
| page_size: 100 | ||
| cursor_value: "{{ response.nextCursor }}" | ||
| stop_condition: "{{ not response.moreDataAvailable }}" | ||
| partition_router: | ||
| type: SubstreamPartitionRouter | ||
| parent_stream_configs: | ||
| - type: ParentStreamConfig | ||
| stream: | ||
| type: DeclarativeStream | ||
| name: applications_for_history | ||
| primary_key: | ||
| - id | ||
| retriever: | ||
| type: SimpleRetriever | ||
| requester: | ||
| type: HttpRequester | ||
| url_base: https://api.ashbyhq.com | ||
| authenticator: | ||
| type: BasicHttpAuthenticator | ||
| username: "{{ config['api_key'] }}" | ||
| password: "{{ config['api_key'] }}" | ||
| path: /application.list | ||
| http_method: POST | ||
| request_body_json: | ||
| createdAfter: "{{ timestamp(config['start_date']) * 1000 }}" | ||
| record_selector: | ||
| type: RecordSelector | ||
| extractor: | ||
| type: DpathExtractor | ||
| field_path: | ||
| - results | ||
| paginator: | ||
| type: DefaultPaginator | ||
| page_token_option: | ||
| type: RequestOption | ||
| inject_into: body_json | ||
| field_name: cursor | ||
| page_size_option: | ||
| type: RequestOption | ||
| inject_into: body_json | ||
| field_name: limit | ||
| pagination_strategy: | ||
| type: CursorPagination | ||
| page_size: 100 | ||
| cursor_value: "{{ response.nextCursor }}" | ||
| stop_condition: "{{ not response.moreDataAvailable }}" | ||
| schema_loader: | ||
| type: InlineSchemaLoader | ||
| schema: | ||
| type: object | ||
| $schema: http://json-schema.org/draft-07/schema# | ||
| additionalProperties: true | ||
| properties: | ||
| id: | ||
| type: string | ||
| parent_key: id | ||
| partition_field: application_id | ||
| extra_fields: | ||
| - - status | ||
| - - createdAt | ||
| transformations: | ||
| - type: AddFields | ||
| fields: | ||
| - type: AddedFieldDefinition | ||
| path: | ||
| - application_id | ||
| value: "{{ stream_partition.application_id }}" | ||
| - type: AddedFieldDefinition | ||
| path: | ||
| - application_status | ||
| value: "{{ stream_slice.extra_fields['status'] }}" | ||
| - type: AddedFieldDefinition | ||
| path: | ||
| - application_created_at | ||
| value: "{{ stream_slice.extra_fields['createdAt'] }}" | ||
| schema_loader: | ||
| type: InlineSchemaLoader | ||
| schema: | ||
| type: object | ||
| $schema: http://json-schema.org/draft-07/schema# | ||
| additionalProperties: true | ||
| properties: | ||
| id: | ||
| type: string | ||
| format: uuid | ||
|
Comment on lines
+1554
to
+1556
Contributor
Author
There was a problem hiding this comment. Deliberate, and I'd like to keep it non-nullable. Worth noting the typing has no effect on whether a null gets through:
Contributor
Author
There was a problem hiding this comment. 🚫 Not fixing — disposition marker for the reply above, which I can't edit in place (the API rejects edits to inline review comments from this account). |
||
| application_id: | ||
| type: | ||
| - "null" | ||
| - string | ||
| application_status: | ||
| type: | ||
| - "null" | ||
| - string | ||
| application_created_at: | ||
| type: | ||
| - "null" | ||
| - string | ||
| format: date-time | ||
| stageId: | ||
| type: | ||
| - "null" | ||
| - string | ||
| format: uuid | ||
| title: | ||
| type: | ||
| - "null" | ||
| - string | ||
| enteredStageAt: | ||
| type: | ||
| - "null" | ||
| - string | ||
| format: date-time | ||
| leftStageAt: | ||
| type: | ||
| - "null" | ||
| - string | ||
| format: date-time | ||
| stageNumber: | ||
| type: | ||
| - "null" | ||
| - integer | ||
| actorId: | ||
| type: | ||
| - "null" | ||
| - string | ||
| allowedActions: | ||
| type: | ||
| - "null" | ||
| - array | ||
| items: | ||
| type: string | ||
| spec: | ||
| type: Spec | ||
| connection_specification: | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch on the inconsistency, but it points the other way:
limitis correct and the existingper_pageusages are the latent bug. Ashby'sApplicationListRequestisadditionalProperties: falsewithsyncToken,cursor,limit,createdAfter,createdBefore,status,jobId, andexpand— there is noper_pageparameter anywhere in the API. So the streams injectingper_pageare most likely having their page size silently ignored today and falling back to Ashby's default page size, which is exactly why this connector's measured throughput is as low as it is.I'm not fixing that here on purpose. Correcting
per_pagetolimitacross the existing streams changes the request shape and page count for every stream in the connector, which deserves its own PR with its own version bump and its own regression run rather than riding along inside a new-stream PR — especially with #84274 open against this same manifest. The new stream uses the documented parameter so it's right from the start, and I've flagged the cleanup separately.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🚫 Not fixing here — filed as #84394 instead. Disposition marker plus the tracking issue for the
per_pagecleanup, which I couldn't add to the reply above (the API rejects edits to inline review comments from this account).