Exporting all the content from a schema

Support
1

Hi Sebastian,

At our organization we use the cloud version of Squidex.
I was asked to export all the data from a schema. The schema has approximately 330k content items.

Before I knew there is a cli to export the data I have written a small script that uses the API. It uses the REST version and contains the following lines:

function exportSchemaData(accessToken, appName, schemaName, skip, take, callback) {
    const options = {
        url: `https://cloud.squidex.io/api/content/${appName}/${schemaName}?$skip=${skip}&$top=${take}`,
        headers: {
            'Authorization': `Bearer ${accessToken}`,
            'Content-Type': 'application/json'
        }
    };

    request.get(options, (error, response, body) => {
        if (error) {
            return callback(error);
        }
        const data = JSON.parse(body);
        callback(null, data);
    });
}

It works quickly in the beginning but then starts to slow down as the skip number grows. Every request returns only 200 items. Around item 60k the server starts to return http 408 request timeout which makes it really hard and slow to export data. I managed to export like 96k items.

Then I found that there is an option to use the cli.
I did the following commands:

sq contents export users --fields=fullName,email,phone,birthdays,appVersion,operatingSystem,lastLogin,termsAndConditions
sq contents export users --format=JSON

Every time on item 101798 I received http 408 request timeout.
In the first case it created a file. In the second it didn’t.

What am I doing wrong? How can I export all the content from this schema?

Thank you in advance,
Tzvi.

2

Database are really bad for doing that, because they have to scan more and more items until the performance dies.

There is a streaming endpoint which provides you all content items in a single database call:

I am not sure how do that with javascript, because it is just one line per content item.