[IMPLEMENTED] Patch Node SDK generator to allow upload Asset file from Readable interface (Node.js, TypeScript)

daball · August 20, 2024, 1:38am

I’ve burned a lot of time up trying to get this one function to work. I get a variety of error messages depending on which way I try to work around it.

I’m trying to download an image file from a URL, store it in memory, and without saving it to disk first, upload it to Squidex as an asset.

Here is my simple algorithm which ought to work, but seems like it doesn’t.

async function uploadDownloadedImageToSquidexAsAsset(downloadUrl: string, assetFolderId: string) {
    let filename = downloadUrl.substring(downloadUrl.lastIndexOf('/')+1);
    let response = await axios.get(downloadUrl, { timeout: TIMEOUT_IN_SECONDS * 1000, responseType: 'arraybuffer' });
    let assetDto = await client.assets.postAsset(Buffer.from(response.data) as any, { timeoutInSeconds: TIMEOUT_IN_SECONDS });
    assetDto = await client.assets.putAsset(assetDto.id, { fileName: filename, metadata: { ...assetDto.metadata, 'amazon-url': downloadUrl }, tags: ['amazon', 'product'] })
    assetDto = await client.assets.putAssetParent(assetDto.id, { parentId: assetFolderId });
    return assetDto;
}

I’ve tried using Readable, Buffer, Stream, ArrayBuffer; I’ve mocked ReadStream the best I can; I’ve used memfs as a workaround which mimics the fs API and sent it a faux File object. Nothing works. I mostly keep getting this same error:

error: Status code: 400
Body: {
  "message": "Validation error",
  "traceId": "00-6bb78641702c77e9686bb142cbd30cb3-b8b1002f4709eecf-01",
  "type": "https://www.rfc-editor.org/rfc/rfc7231#section-6.5.1",
  "details": [
    "Can only upload one file."
  ],
  "statusCode": 400
}

The thing is I’m only uploading one file.

At one point I was going to try to override or otherwise extend the implementation of postAsset(file: File|fs.ReadStream, requestOptions?: Assets.RequestOptions) so I could force it to do something different, but the complication I ran into extending the Asset class was the protected member I wanted to copy from the existing Asset class, and it is also hidden.

Short of forking the whole node-sdk for Squidex to make it do my bidding, I’m not sure how to do this without first saving a file into a temporary folder and then deleting it immediately afterwards.

I don’t know if this is a bug or intentional. I just know I’ve spent a lot of hours doing nothing but throwing variously shaped objects at postAsset() without getting anything done.

The only time I got any other error it complained that it couldn’t stat the file (that doesn’t exist because I completely lied about it existing in the first place). I suppose that’s because it was trying to use Node fs functions instead of the ones I gave it.

This just can’t be the only way to do this. There has to be a better way. Or I guess I can perhaps fork it and then ensure there’s a way. Do you want me to work on this function spec and maybe give you a pull request or something? Because I think I ought to be able to just give you a file in memory and have the SDK roll with it.

daball · August 20, 2024, 3:28am

So I went ahead and forked the code and made a few lines of changes that fix this problem I’m having with the SDK. I have initiated pull request # 56 for this change. See feat: support for uploading assets from memory by daball · Pull Request #56 · Squidex/sdk-node (github.com) for details on the code changes requested to the Node SDK.

Then the fix for my previously broken code now looks like this with the small adjustment to the SDK code:

export async function uploadDownloadedImageToSquidexAsAsset(downloadUrl: string, assetFolderId: string) {
    let filename = downloadUrl.substring(downloadUrl.lastIndexOf('/')+1);
    let response = await axios.get(downloadUrl, { timeout: TIMEOUT_IN_SECONDS * 1000, responseType: 'arraybuffer' });
    let assetDto = await client.assets.postAsset({ readable: response.data, fileName: filename }, { timeoutInSeconds: TIMEOUT_IN_SECONDS });
    assetDto = await client.assets.putAsset(assetDto.id, { fileName: filename, metadata: { ...assetDto.metadata, 'amazon-url': downloadUrl }, tags: ['amazon', 'product'] })
    assetDto = await client.assets.putAssetParent(assetDto.id, { parentId: assetFolderId });
    return assetDto;
}

Particular emphasis on this line, which presents how you would handle uploading the asset if it were already in memory:

    let assetDto = await client.assets.postAsset({ readable: response.data, fileName: filename }, { timeoutInSeconds: TIMEOUT_IN_SECONDS });

Sebastian · August 20, 2024, 6:06am

Thanks a lot. The problem is that the code is auto generated. So I cannot accept the pull request. I have to check the status of the SDK generator and in the worst case switch to another one.

daball · August 20, 2024, 6:21am

I have no idea how it is generated, but perhaps you can look upstream on that. It’s kind of odd to expect a file to be there 100% of the time, from whatever artifacts it is generated from. It’s quite alright if you can’t accept the pull request. Hopefully you can do some upstream changes that help produce a better generated artifact. Either way, I’m keeping my fork and I can merge your upstream changes in as needed. And if anyone else should encounter this same problem, that commit will have the workaround. Thanks!

Sebastian · August 20, 2024, 8:14am

It needs to be fixed. 100%. But your fix would be lost when I generate the SDK again. I am working to get the generation fixed and when I use the newer version, I will see what is generated.

Then there are 2 potential solutions:

Report this as a feature request.
Write some post-hook that modifies the code after generation and applies your fix.

We should definitely both (because I don’t want to wait for the feature to be implemented). I will update you about the status. Right now I am running into a few bugs with the generator that I have already reported.

daball · August 20, 2024, 10:53pm

Feature Request

I have moved this post to the Feature Requests section.

Consider this patch in the pull request as a feature request for the Node SDK code generator. Comparing Squidex:main…daball:memory-assets · Squidex/sdk-node (github.com).

Through whatever mechanism that exists where the Node SDK produces File|fs.ReadStream it actually needs a File|fs.ReadStream|{ readable: stream.Readable|File|fs.ReadStream, fileName: string} instead and that in those three functions it needs to see if you have provided the {readable, fileName} interface because the way you use the upstream FormData object changes when you are loading memory files versus asking FormData to load the file and then read it.

If you inspect the code for FormData, on line 52 we see that you can also simply provide the fileName as a string rather than the options object, which was how I decided to use it in my patch. Their source code:

///Starting at line 52
FormData.prototype.append = function(field, value, options) {

  options = options || {};

  // allow filename as single option
  if (typeof options == 'string') {
    options = {filename: options};
  }
//...

I chose to use an interface in place with the current spec because it doesn’t break any old code while providing both features. Others may find many similar solutions, but this was the one I liked because it keeps all old code identical while enabling the ability to load a file from memory instead of disk.

You may find that would prefer to change the interface name or its fields to something similar but different. In that case I would convert my code to match your interface as long as it provides the necessary function. For instance you may prefer to call the option field file instead of readable, etc.

daball · August 25, 2024, 4:24am

I found a few sections of upstream code related to fern which looks suspiciously similar to the code I’m complaining about in the generated artifacts.

fern/seed/ts-sdk/file-upload/wrap-file-properties/src/api/resources/service/client/requests/MyRequest.ts at main · fern-api/fern (github.com)

fern/seed/ts-sdk/file-upload/wrap-file-properties/src/api/resources/service/client/requests/JustFileRequet.ts at main · fern-api/fern (github.com)

fern/seed/ts-sdk/file-upload/wrap-file-properties/src/api/resources/service/client/Client.ts at main · fern-api/fern (github.com)

daball · August 25, 2024, 4:43am

Actually I’m looking right here at a Pull Request/Merge that happened back on July 16, 2024.

fern/seed/ts-sdk/file-upload/wrap-file-properties/snippet.json at main · fern-api/fern (github.com)

According to this, the problem has been solved. It’s not important that I can use it with a Buffer, per se. The reason it fails without a File is because I think a file name might be required when handling a multipart form file upload. When the file name isn’t present the server rejects it, or something like that.

In this pull, they said you can accept a Blob as a file parameter, which supports sending the filename with the request. For instance, one of the options for the HTTP client, axios, is that you can request the file download data as a blob.

When I put the pull request in for my patch, which we’ve discussed now why you can’t accept it, I’m showing the last pull request you’ve merged in was back on Oct 4, 2023.

Squidex/sdk-node: Node.js SDK for the Squidex API (github.com)

Therefore, I suspect that if you were upgrade the upstream fern dependency and rebuild the SDK we might find that the generated code from the recent updates to the upstream fern will contain Blob as an option. That would fix this problem I believe.

Sebastian · August 25, 2024, 9:26am

Fern is a commercial product. I was an early adaptor and the main reason was that I think the generated APIs work really good, especially for discriminators.

The main version is a cloud product and the generators are actually executed on their cloud. Just two days ago they fixed their local run mode that is based on docker. But it is very buggy. So basically I see two options:

Report bugs as much as possible and hope that fern will be fixed. Obviously the local version is not their main product, so it could take a while.
Switch to another generator. The generated API is probably not as good and you will have to live with breaking changes. To be honest I tend towards this option, because I lost a little bit of trust in fern

Sebastian · August 25, 2024, 3:46pm

I have updated the SDK to open-api-generator: GitHub - Squidex/sdk-node at openapi-generator

Would like to have your feedback here. The API is not 100% the same, but I tried to make it as close as possible to the existing interface.

daball · August 25, 2024, 8:39pm

My first observation is that the package hasn’t been updated on NPM.

@squidex/squidex - npm (npmjs.com)

This means I’ll need to sideload it somehow. Due to limitations with Vite I won’t be able to use npm link for this task (at least in Windows). I will use this workaround for testing.

cd node_modules
cd "@squidex"
mv squidex squidex-npm
git clone https://github.com/Squidex/sdk-node.git
mv sdk-node squidex
cd squidex
git checkout openapi-generator
git pull
npm run build

I will note that we have 427 build errors being emitted from the TypeScript transpiler. Most are declared unused variable on the function.

...
Found 427 errors in 203 files.

Errors  Files
     1  src/generated/models/ActionTypeEnum.ts:43
     2  src/generated/models/AddFieldDto.ts:63
     2  src/generated/models/AddLanguageDto.ts:43
     2  src/generated/models/AddRoleDto.ts:43
     2  src/generated/models/AddWorkflowDto.ts:43
     1  src/generated/models/AlgoliaRuleActionDto.ts:90
     3  src/generated/models/AllContentsByPostDto.ts:70
     3  src/generated/models/AnnotateAssetDto.ts:58
     2  src/generated/models/AppDto.ts:138
     2  src/generated/models/AppLanguageDto.ts:85
     3  src/generated/models/AppLanguagesDto.ts:63
     4  src/generated/models/AppSettingsDto.ts:97
     1  src/generated/models/ArrayCalculatedDefaultValue.ts:42
     2  src/generated/models/ArrayFieldPropertiesDto.ts:67
     2  src/generated/models/AssetChangedRuleTriggerDto.ts:41
     2  src/generated/models/AssetDto.ts:229
     2  src/generated/models/AssetFolderDto.ts:78
     1  src/generated/models/AssetFolderScope.ts:43
     4  src/generated/models/AssetFoldersDto.ts:77
     2  src/generated/models/AssetMeta.ts:43
     1  src/generated/models/AssetPreviewMode.ts:43
     2  src/generated/models/AssetScriptsDto.ts:99
     3  src/generated/models/AssetsDto.ts:70
     2  src/generated/models/AssetsFieldPropertiesDto.ts:177
     1  src/generated/models/AssetType.ts:44
     2  src/generated/models/AssignContributorDto.ts:55
     2  src/generated/models/AuthSchemeDto.ts:77
     2  src/generated/models/AuthSchemeResponseDto.ts:62
     3  src/generated/models/AuthSchemeValueDto.ts:41
     1  src/generated/models/AzureQueueRuleActionDto.ts:75
     2  src/generated/models/BackupJobDto.ts:99
     3  src/generated/models/BackupJobsDto.ts:63
     1  src/generated/models/BooleanFieldEditor.ts:42
     2  src/generated/models/BooleanFieldPropertiesDto.ts:67
     2  src/generated/models/BulkResultDto.ts:69
     4  src/generated/models/BulkUpdateAssetsDto.ts:59
     3  src/generated/models/BulkUpdateAssetsJobDto.ts:97
     1  src/generated/models/BulkUpdateAssetType.ts:43
     3  src/generated/models/BulkUpdateContentsDto.ts:94
     3  src/generated/models/BulkUpdateContentsJobDto.ts:115
     1  src/generated/models/BulkUpdateContentType.ts:48
     2  src/generated/models/CallsUsageDtoDto.ts:106
     2  src/generated/models/CallsUsagePerDateDto.ts:64
     3  src/generated/models/ChangeCategoryDto.ts:34
     2  src/generated/models/ChangePlanDto.ts:43
     2  src/generated/models/ChangeStatusDto.ts:55
     2  src/generated/models/ClientDto.ts:98
     3  src/generated/models/ClientsDto.ts:63
     1  src/generated/models/CommentRuleActionDto.ts:67
     2  src/generated/models/CommentRuleTriggerDto.ts:41
     2  src/generated/models/ComponentFieldPropertiesDto.ts:41
     2  src/generated/models/ComponentsFieldPropertiesDto.ts:73
     4  src/generated/models/ConfigureFieldRulesDto.ts:41
     3  src/generated/models/ConfigureUIFieldsDto.ts:40
     3  src/generated/models/ContentChangedRuleTriggerDto.ts:80
     3  src/generated/models/ContentDto.ts:187
     4  src/generated/models/ContentsDto.ts:83
     2  src/generated/models/ContributorDto.ts:77
     3  src/generated/models/ContributorsDto.ts:82
     2  src/generated/models/ContributorsMetadata.ts:43
     2  src/generated/models/CreateAppDto.ts:49
     2  src/generated/models/CreateAssetFolderDto.ts:49
     2  src/generated/models/CreateClientDto.ts:43
     1  src/generated/models/CreateContentRuleActionDto.ts:84
     2  src/generated/models/CreateRuleDto.ts:63
     4  src/generated/models/CreateSchemaDto.ts:143
     2  src/generated/models/CreateTeamDto.ts:43
     2  src/generated/models/CreateUserDto.ts:64
     2  src/generated/models/CurrentStorageDto.ts:50
     1  src/generated/models/DateTimeCalculatedDefaultValue.ts:42
     1  src/generated/models/DateTimeFieldEditor.ts:42
     2  src/generated/models/DateTimeFieldPropertiesDto.ts:91
     1  src/generated/models/DiscourseRuleActionDto.ts:105
     2  src/generated/models/EditorDto.ts:50
     1  src/generated/models/ElasticSearchRuleActionDto.ts:96
     1  src/generated/models/EmailRuleActionDto.ts:116
     2  src/generated/models/ErrorDto.ts:74
     2  src/generated/models/EventConsumerDto.ts:90
     3  src/generated/models/EventConsumersDto.ts:63
     1  src/generated/models/FastlyRuleActionDto.ts:68
     2  src/generated/models/FeatureDto.ts:50
     3  src/generated/models/FeaturesDto.ts:57
     3  src/generated/models/FieldDto.ts:117
     1  src/generated/models/FieldRuleAction.ts:43
     2  src/generated/models/FieldRuleDto.ts:65
     1  src/generated/models/GeolocationFieldEditor.ts:41
     2  src/generated/models/GeolocationFieldPropertiesDto.ts:49
     2  src/generated/models/HistoryEventDto.ts:78
     1  src/generated/models/ImageFormat.ts:48
     2  src/generated/models/ImportContentsDto.ts:62
     3  src/generated/models/JobDto.ts:126
     2  src/generated/models/JobLogMessageDto.ts:50
     3  src/generated/models/JobsDto.ts:63
     1  src/generated/models/JobStatus.ts:45
     2  src/generated/models/JsonFieldPropertiesDto.ts:41
     2  src/generated/models/LanguageDto.ts:57
     3  src/generated/models/LogDownloadDto.ts:34
     3  src/generated/models/ManualRuleTriggerDto.ts:35
     1  src/generated/models/MediumRuleActionDto.ts:105
     3  src/generated/models/MoveAssetDto.ts:34
     3  src/generated/models/MoveAssetFolderDto.ts:34
     2  src/generated/models/NestedFieldDto.ts:98
     1  src/generated/models/NotificationRuleActionDto.ts:82
     1  src/generated/models/NumberFieldEditor.ts:44
     2  src/generated/models/NumberFieldPropertiesDto.ts:91
     1  src/generated/models/OpenSearchRuleActionDto.ts:96
     2  src/generated/models/PatternDto.ts:56
     3  src/generated/models/PlanChangedDto.ts:34
     2  src/generated/models/PlanDto.ts:109
     3  src/generated/models/PlansDto.ts:95
     1  src/generated/models/PlansLockedReason.ts:44
     1  src/generated/models/PrerenderRuleActionDto.ts:68
     3  src/generated/models/QueryDto.ts:52
     3  src/generated/models/QueryJsonDto.ts:89
     1  src/generated/models/ReferencesFieldEditor.ts:46
     2  src/generated/models/ReferencesFieldPropertiesDto.ts:103
     2  src/generated/models/ReferralInfo.ts:57
     2  src/generated/models/RenameAssetFolderDto.ts:43
     2  src/generated/models/RenameTagDto.ts:43
     2  src/generated/models/ReorderFieldsDto.ts:43
     1  src/generated/models/ResizeMode.ts:47
     2  src/generated/models/Resource.ts:50
     2  src/generated/models/ResourceLink.ts:56
     2  src/generated/models/ResourcesDto.ts:50
     2  src/generated/models/RestoreJobDto.ts:79
     2  src/generated/models/RestoreRequestDto.ts:49
     2  src/generated/models/RichTextFieldPropertiesDto.ts:89
     2  src/generated/models/RoleDto.ts:92
     3  src/generated/models/RolesDto.ts:63
     2  src/generated/models/RuleDto.ts:152
     3  src/generated/models/RuleElementDto.ts:88
     2  src/generated/models/RuleElementPropertyDto.ts:92
     2  src/generated/models/RuleEventDto.ts:125
     3  src/generated/models/RuleEventsDto.ts:70
     1  src/generated/models/RuleFieldEditor.ts:49
     1  src/generated/models/RuleJobResult.ts:45
     1  src/generated/models/RuleResult.ts:44
     3  src/generated/models/RulesDto.ts:69
     2  src/generated/models/ScheduleJobDto.ts:71
     2  src/generated/models/SchemaChangedRuleTriggerDto.ts:41
     2  src/generated/models/SchemaCondition.ts:49
     4  src/generated/models/SchemaDto.ts:208
     2  src/generated/models/SchemaPropertiesDto.ts:91
     3  src/generated/models/SchemaScriptsDto.ts:64
     3  src/generated/models/SchemasDto.ts:63
     1  src/generated/models/SchemaType.ts:43
     1  src/generated/models/ScriptRuleActionDto.ts:60
     2  src/generated/models/SearchResultDto.ts:78
     1  src/generated/models/SearchResultType.ts:46
     1  src/generated/models/SignalRRuleActionDto.ts:105
     2  src/generated/models/SimulatedRuleEventDto.ts:102
     3  src/generated/models/SimulatedRuleEventsDto.ts:70
     1  src/generated/models/SkipReason.ts:51
     1  src/generated/models/SlackRuleActionDto.ts:68
     2  src/generated/models/SortNode.ts:59
     1  src/generated/models/SortOrder.ts:42
     2  src/generated/models/StatusInfoDto.ts:50
     2  src/generated/models/StorageUsagePerDateDto.ts:57
     1  src/generated/models/StringContentType.ts:43
     1  src/generated/models/StringFieldEditor.ts:50
     2  src/generated/models/StringFieldPropertiesDto.ts:169
     5  src/generated/models/SynchronizeSchemaDto.ts:119
     1  src/generated/models/TagsFieldEditor.ts:43
     2  src/generated/models/TagsFieldPropertiesDto.ts:85
     2  src/generated/models/TeamDto.ts:91
     2  src/generated/models/TemplateDetailsDto.ts:57
     2  src/generated/models/TemplateDto.ts:78
     3  src/generated/models/TemplatesDto.ts:63
     3  src/generated/models/TransferToTeamDto.ts:34
     2  src/generated/models/TranslateDto.ts:56
     2  src/generated/models/TranslationDto.ts:65
     1  src/generated/models/TranslationStatus.ts:46
     1  src/generated/models/TweetRuleActionDto.ts:76
     1  src/generated/models/TypesenseRuleActionDto.ts:90
     1  src/generated/models/UIFieldEditor.ts:41
     2  src/generated/models/UIFieldPropertiesDto.ts:49
     3  src/generated/models/UpdateAppDto.ts:40
     4  src/generated/models/UpdateAppSettingsDto.ts:75
     3  src/generated/models/UpdateAssetScriptsDto.ts:70
     3  src/generated/models/UpdateClientDto.ts:58
     2  src/generated/models/UpdateFieldDto.ts:50
     3  src/generated/models/UpdateLanguageDto.ts:46
     3  src/generated/models/UpdateProfileDto.ts:34
     2  src/generated/models/UpdateRoleDto.ts:49
     3  src/generated/models/UpdateRuleDto.ts:65
     3  src/generated/models/UpdateSchemaDto.ts:70
     2  src/generated/models/UpdateSettingDto.ts:43
     2  src/generated/models/UpdateTeamDto.ts:43
     2  src/generated/models/UpdateUserDto.ts:63
     2  src/generated/models/UpdateWorkflowDto.ts:69
     5  src/generated/models/UpsertSchemaDto.ts:107
     3  src/generated/models/UpsertSchemaFieldDto.ts:93
     2  src/generated/models/UpsertSchemaNestedFieldDto.ts:75
     1  src/generated/models/UsageRuleTriggerDto.ts:67
     2  src/generated/models/UserDto.ts:85
     2  src/generated/models/UserProperty.ts:50
     3  src/generated/models/UsersDto.ts:70
     1  src/generated/models/WebhookMethod.ts:45
     1  src/generated/models/WebhookRuleActionDto.ts:104
     2  src/generated/models/WorkflowDto.ts:89
     3  src/generated/models/WorkflowsDto.ts:70
     3  src/generated/models/WorkflowStepDto.ts:71
     3  src/generated/models/WorkflowTransitionDto.ts:40
Files:                         452
Lines of Library:            39157
Lines of Definitions:       129518
Lines of TypeScript:         33489
Lines of JavaScript:             0
Lines of JSON:                   0
Lines of Other:                  0
Identifiers:                180509
Symbols:                    107323
Types:                        8853
Instantiations:               4374
Memory used:               341663K
Assignability cache size:     2390
Identity cache size:           158
Subtype cache size:             88
Strict subtype cache size:     120
I/O Read time:               0.05s
Parse time:                  0.51s
ResolveModule time:          0.06s
ResolveTypeReference time:   0.01s
ResolveLibrary time:         0.01s
Program time:                0.72s
Bind time:                   0.34s
Check time:                  0.69s
transformTime time:          0.21s
commentTime time:            0.14s
I/O Write time:              0.14s
printTime time:              0.93s
Emit time:                   0.93s
Total time:                  2.68s

When I run npm run astro build on my project with the release candidate SDK, I’m getting a couple of errors here:

Debugger listening on ws://127.0.0.1:54501/cff097e3-ec4e-4702-91c4-274c4e3f40ca
For help, see: https://nodejs.org/en/docs/inspector
Debugger attached.

> dashersupply@0.2.1 astro
> astro build

Debugger listening on ws://127.0.0.1:54504/c0ede1cc-1f1e-4469-84f1-d3bceb054dc1
For help, see: https://nodejs.org/en/docs/inspector
Debugger attached.
16:05:53 [WARN] [config] The feature "astro:env getSecret" is experimental and subject to change (used by @astrojs/node).
16:05:53 [vite] Re-optimizing dependencies because lockfile has changed
16:05:53 [types] Generated 101ms
16:05:53 [build] output: "server"
16:05:53 [build] directory: D:\sources\dashersupply\dist\
16:05:53 [build] adapter: @astrojs/node
16:05:53 [build] Collecting build info...
16:05:53 [build] ✓ Completed in 125ms.
16:05:53 [build] Building server entrypoints...
16:05:54 [ERROR] [vite] x Build failed in 411ms
[commonjs--resolver] Failed to resolve entry for package "@squidex/squidex". The package may have incorrect main/module/exports specified in its package.json.
  Stack trace:
    at packageEntryFailure (file:///D:/sources/dashersupply/node_modules/vite/dist/node/chunks/dep-Cy9twKMn.js:46533:15)
    at tryNodeResolve (file:///D:/sources/dashersupply/node_modules/vite/dist/node/chunks/dep-Cy9twKMn.js:46346:16)
    at Object.handler (file:///D:/sources/dashersupply/node_modules/vite/dist/node/chunks/dep-Cy9twKMn.js:65535:15)
    at async PluginDriver.hookFirstAndGetPlugin (file:///D:/sources/dashersupply/node_modules/rollup/dist/es/shared/node-entry.js:19813:28)
    at async ModuleLoader.resolveId (file:///D:/sources/dashersupply/node_modules/rollup/dist/es/shared/node-entry.js:18890:15)
Waiting for the debugger to disconnect...
Waiting for the debugger to disconnect...

Upon inspection the package.json file shows the entry point and the types are at ‘./index.js’ and ‘./index.d.ts’, respectively, but those files are not present. Instead they should be at ‘./dist/index.js’ and ‘./dist/index.d.ts’ instead. I’ve patched the package.json for the SDK, and then try again.

Re-run npm run-script astro build after patching node_modules/@squidex/squidex/package.json with success.

Looks like the new interface supports Blobs, so I think this new SDK will resolve this issue.

The new upload function should work like this with the new SDK.

export async function uploadDownloadedImageToSquidexAsAsset(downloadUrl: string, assetFolderId: string) {
    let filename = downloadUrl.substring(downloadUrl.lastIndexOf('/')+1);
    let response = await axios.get(downloadUrl, { timeout: TIMEOUT_IN_SECONDS * 1000, responseType: 'blob' });
    let assetDto = await client.assets.postAsset({ file: response.data, fileName: filename });
    assetDto = await client.assets.putAsset({ id: assetDto.id, annotateAssetDto: { metadata: { ...assetDto.metadata, 'amazon-url': downloadUrl }, tags: ['amazon', 'product'] } })
    assetDto = await client.assets.putAssetParent({ id: assetDto.id, moveAssetDto: { parentId: assetFolderId } });
    return assetDto;
}

Here is a list of all of the things as an SDK consumer I had to change in my project, aside from the package.json patch, as a result of the migration.

Migrated to release candidate Squidex SDK · 77b83f51dd - dashersupply - Gitea: Git with a cup of tea (daball.me)

I will note that the new SDK also fails at getReferences(), which you were saying needed a patch in .NET codebase for Squidex.

		let redirectReferencesDto = await client.contents.getReferences({ schema: SCHEMAS.REDIRECTS, id: redirectDto.items[0].id });
		console.log("redirectReferencesDto:", redirectReferencesDto);

I didn’t figure out which version I need to set my Docker to get the patch for that on the .NET side.

TypeError: Cannot read properties of undefined (reading 'map')
    at ContentsDtoFromJSONTyped (...\node_modules\@squidex\squidex\dist\generated\models\ContentsDto.js:49:39)
    at ContentsDtoFromJSON (...\node_modules\@squidex\squidex\dist\generated\models\ContentsDto.js:39:12)
    at JSONApiResponse.transformer (...\node_modules\@squidex\squidex\dist\generated\apis\ContentsApi.js:532:105)
    at JSONApiResponse.<anonymous> (...\node_modules\@squidex\squidex\dist\generated\runtime.js:298:25)
    at Generator.next (<anonymous>)
    at fulfilled (...\node_modules\@squidex\squidex\dist\generated\runtime.js:18:58)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Fine work, Sebastian. I’ll let you know if I have any trouble, but I suspect this SDK will work even better than the fern SDK. I’m getting better response times.

daball · August 26, 2024, 12:57am

After a great deal of struggling with the new SDK format, I finally was able to get the passthrough upload-my-download to work. I could only get it after constructing my own HTTP body and handing it over. It would not work correctly otherwise.

export async function uploadDownloadedImageToSquidexAsAsset(downloadUrl: string, assetFolderId: string) {
    let filename = downloadUrl.substring(downloadUrl.lastIndexOf('/')+1);//.replace(/[A-Z0-9\.-_]*, '$0');
    let response = await axios.get(downloadUrl, { timeout: TIMEOUT_IN_SECONDS * 1000, responseType: 'arraybuffer' });
    let form = new FormData();
    let blob = new Blob(response.data, {type: mimeLookup(filename) as string});
    form.append('fileName', filename);
    form.append('file', blob, filename);
    form.append('parentId', assetFolderId);
    let assetDto = await client.assets.postAsset({ file: blob, fileName: filename, fileUrl: filename, parentId: assetFolderId }, {
        body: form
    });
    assetDto = await client.assets.putAsset({ id: assetDto.id, annotateAssetDto: { metadata: { ...assetDto.metadata, 'amazon-url': downloadUrl }, tags: ['amazon', 'product'] } })
    return assetDto;
}

Do you think this will be the direction of the project moving forward? Because I’m ready to merge this commit into my main tree.

Sebastian · August 26, 2024, 5:02am

It is still WIP.

This is basically a result of the code generation. I have disabled the typescript settings noUnusedParameters for my build.

This has actually been fixed in the code.

Have you had a look to the test. This is what I do to upload a file:

const fileStream = await fs.readFile("tests/assets/logo-wide.png");
const fileBlob = new File([fileStream], "logo-wide.png", { type: "image/png" });

const createdAsset = await client.assets.postAsset({ file: fileBlob });

Squidex can also download the file directly in newer versions:

const fileName = downloadUrl.substring(fileUrl.lastIndexOf('/')+1);//.replace(/[A-Z0-9\.-_]*, '$0');

const createdAsset = await client.assets.postAsset({ fileUrl, fileName });

Probably. I am still solving small issues. For example I fix some namings and so on.

Sebastian · August 26, 2024, 6:07pm

I have managed to fix the compilation errors. I just run eslint after code generation … Do you like the API / Client Design? Anything else to fix?

daball · August 26, 2024, 6:32pm

I had not used the test for reference, however I was trying not to save a temporary file. I did test it two different ways:

await client.assets.postAsset({ file: new Blob(arrayBuffer), fileName: 'filename' });

^In this case it was the global Node.js Blob and not the one in node:buffer. The one in node:buffer was incompatible with the interface expected by postAsset().

And:

await client.assets.postAsset({ fileUrl: 'url', fileName: 'filename'})

Both threw an error about you can only upload one file. Finally just setting the body of my request according to the form-data/FormData class was what finally worked for me using a file in memory.

Because I wasn’t able to get Squidex to do the downloading for me, I suspect I need to switch to a different build. My Docker Compose file is currently set to image: "squidex/squidex:7" and image: "mongo:6".

A quick look at Docker Hub, I see that you frequently post updates. For instance, 3 days ago you posted an update to dev along with a new image tagged dev-8064 whereas the version 7 I’m running which is also the same as 7.14.0 was released 2 months ago. Are the dev builds stable for the most part? I would want to do a backup right before any migration.

I have managed to fix the compilation errors. I just run eslint after code generation … Do you like the API / Client Design? Anything else to fix?

I do like the new design. I haven’t used all of the features yet to be able to tell you if anything else is broken. The main issue with using it as-is was that the package.json points to files that aren’t there. Other than that, I was able to use everything I’m using now other than getReferences(), with minimal changes. The changes I did make I honestly wondered why it wasn’t structured that way sooner. I think the new function interfaces are a step in the right direction, and especially if it means you can iterate without relying on a proprietary algorithm. I also found that the ability to manually override the body, for instance, was important for working around any assumptions that may be made by the generated code without writing an entire REST client from scratch.

Perhaps to aid in migration there could be a shim that converts calls of the old interface to calls for the new interface. It was no big deal for me to fix mine, but maybe others may have a different experience. And if you do provide a shim, then you could maybe have some way to version them side-by-side so that you could use the shim (e.g., v1-compat, etc.) or the new interfaces (e.g., v2) perhaps side-by-side in the same file, with the goal of eventually deprecating the v1 API. Like if v2 was opt-in it wouldn’t break anybody’s code if the shim was functionally the same as the fern generated code. It might break my code at this point, but if opting in were as simple as using a Node sub-module within the same library, e.g., in a ./packages folder like this one, so that devs can opt-in like import { Squidex } from '@squidex/squidex-v2' or something similar, I could see that working with minimal interruption even for me. At that point you begin documenting the new API and throwing deprecation warnings and everyone’s either using the original fern API or the API shim and perhaps completly oblivious about it. I’m not sure, this is your call I guess. I like the new one. But if you break a bunch of people’s code you’re probably going to hear about it eventually.

For instance, one such shim might look like this:

import { SquidexClient as SquidexClientV2 } from "@squidex/squidex-v2";

export class SquidexAssetsV1Shim {
  client: SquidexClientV2; //or however you achieve this reference

  protected convertGetAssetOptions(oldOptions) {
    return oldOptions; //transform the oldOptions into the new ones
  }

  public async getAsset (id, options) {
    return await client.assets.getAsset({ id }, this.convertGetAssetOptions(options)); //or whatever is accurate
  }
} // exported as fern-like replacement

Before going all-in on making a bunch of shims, you can start off easy by simply including the current fern version as well as the new version with the opt-in interface and then gradually replace one then another with shims as you phase out the old operations and replace them with the new ones. This gives you several advantages. You let developers segregate themselves into their own A-B camps for A-B testing. As you gradually build the shim replacement, you can sprinkle in your own new features such as the one I originally requested at the top of this thread, no longer relying upon the generator that you are phasing out. As each shim release can be incremental, you can therefore get immediate feedback if something broke in the shim, setting your own pace for the eventual complete delivery of a final shim which removes the fern dependency entirely. And if everything eventually works with the shim you can finally replace all of the code, even if nobody notices all of this work is happening underneath the hood all along the way.

Sebastian · August 26, 2024, 7:42pm

There was a bug in the OpenAPI spec. The parameters had the wrong name. I added a test for that which works after my fix:

it("should upload and fetch asset from url", async () => {
    const fileName = "logo.png";
    const fileUrl  = "https://raw.githubusercontent.com/Squidex/squidex/master/media/logo-squared.png";

    const createdAsset = await client.assets.postAsset({ name: fileName, url: fileUrl });

    const asset = await client.assets.getAsset({ id: createdAsset.id });
    expect(asset.id).toEqual(createdAsset.id);
    expect(asset.fileName).toEqual(fileName);
    expect(asset.fileSize).toBeGreaterThan(100);
    expect(asset.mimeType).toEqual("image/png");
});

A lot of bugs in the openapi-generator. I made some experience how to fix them when I have built the PHP SDK, but I still needed a few fixes here and there. Thankfully it is possible, because I can override the templates.

Good idea in general, but I think too much work. There are 100+ API methods. I am not sure if I have the time for that at the moment.

daball · August 26, 2024, 7:46pm

I edited and suggested a more modest approach than a full shim. I.e. Phase in the development of the shim as you gradually phase out the v1. Side-by-side is perhaps the best way not to break anyone else’s code. I think you would use the fern output as the starting point until all that is left are shims. Which provides plenty of time to space out the development of the shim. And you could prioritize which aspects to convert as you run up against issues with the v1 API, targeting those areas which are most problematic first.

I recently did a JavaScript to TypeScript conversion for a very old API and it took all of my time for several days and there were definitely a lot of files. And now it works and the type-hinting is great. Maybe it isn’t perfect though. I haven’t worked out all of the bugs maybe.

I transcribed this Unzipped Amazon’s Product Advertising API SDK example… · daball/amazon-pa-api-5.0-sdk-node-ts@62542b8 (github.com) into daball/amazon-pa-api-5.0-sdk-node-ts (github.com). It was very tedious. And it may not work as well as the original. The main difference for me was that I could read it after I was done. The original code was hard to read and TypeScript wanted type annotations and so did I, frankly. I rewrote every single file until I was done.

Starting with side-by-side with no shim working towards eventually a full shim is the alternative to gung-ho.

Sebastian · August 26, 2024, 8:45pm

I have adjusted the generator template to make the required parameters as normal method parameters. I think this was the default behavior before. The differences are very small now.

daball · August 27, 2024, 12:16am

Alright I’ve merged in the latest. I’m seeing that these two files are missing from the build. And I had to adjust some of my code back. See this commit: fix: Some of the release candidate API interface changes were reverted on the upstream. · eb4e0f7f2e - dashersupply - Gitea (daball.me)

{
    "name": "@squidex/squidex",
    "version": "1.2.1",
...
    "main": "./index.js", //--> "./dist/index.js"
    "types": "./index.d.ts", //--> "./dist/index.d.ts"
...

And I am happy to report I’ve finally my upload function is now working correctly.

// URL and File are both Web API globals, not to be mixed up with any of the Node API types
// See: https://developer.mozilla.org/en-US/docs/Web/API/File
// See: https://developer.mozilla.org/en-US/docs/Web/API/URL
// I used https://www.npmjs.com/package/mime-types for the MIME type lookup
export async function uploadDownloadedImageToSquidexAsAsset(downloadUrl: string, assetFolderId: string) {
    let url = new URL(downloadUrl);
    let filename = url.pathname.substring(url.pathname.lastIndexOf('/')+1);
    let res = await fetch(downloadUrl);
    let ab = await res.arrayBuffer();
    let file = new File([ab], filename, { type: mimeLookup(filename) as string });
    let assetDto = await client.assets.postAsset({ file, parentId: assetFolderId });
    assetDto = await client.assets.putAsset(assetDto.id, { metadata: { ...assetDto.metadata, 'amazon-url': downloadUrl }, tags: ['amazon', 'product'] });
    return assetDto;
}

Sebastian · August 27, 2024, 3:10pm

I have published it to NPM