Create (if not exists) or Update Content (if exists) rule

[daball]

The CreateContent rule almost does exactly what it needs to do.

I have a set of collections like 'page', 'product', 'brand', 'marketplace', etc., and each of them have a slug field. Those slugs don’t follow a URL routing convention. They share the same URL namespace.

Collisions aside (which are definitely possible), I would like a lookup collection like 'slugs' that I can query to find out which collection I can lookup for the slug to get data needed on the frontend to send it to the right route handler.

So I understand that to do that I need a rule. I would like to use a CreateContent rule but if the event I have is type == 'Updated' I don’t want to keep creating a new content item over and over again.

So perhaps I am looking for a CreateIfNotExistsOrUpdateIfExistsContent rule, which would be a much nicer rule.

I would be okay with using a Run Script rule but I couldn’t edit it. So I sent in a bug report for that already.

For my feature workaround, a Webhook is the only way I can see solving my issue without substantially modifying the Squidex code.

And the whole project needs more docs. But other than that I really like what you’ve done with it.

[daball]

And I guess I realized that even this won’t necessarily solve all the caveats of my problem because the slug could change and the old slug might be sticky on the cache table. I may look into using redis or something I can just query quickly. So maybe this feature isn’t really all that important for me.

It might be better to better to describe the feature I need as a “one-way sync content with projection” rule rather than create (if not exists) or update (if exists). I rescind my original feature request because it doesn’t fully solve my problem. But do know this is a thing you could possibly implement and that would be pretty cool.

[Sebastian]

@daball

And the whole project needs more docs. But other than that I really like what you’ve done with it.

True.

@daball

It might be better to better to describe the feature I need as a “one-way sync content with projection” rule rather than create (if not exists) or update (if exists). I rescind my original feature request because it doesn’t fully solve my problem. But do know this is a thing you could possibly implement and that would be pretty cool.

For an create or update feature you need a unique identifier to identify whether two objects are the same. The ID might work (you can also use custom IDs, but they need to be globally unique). But if the ID is not enough there needs to be more logic around that and I am not not sure if this can be provided easily in rules.

There is a bulk endpoint in the API that provides this feature. You can define a filter to identity your object.

[daball]

So I will show you the code that worked for me. I have a few helpers to help along the way. I’ve put this in my frontend and I’m probably going to use a back door port on the inside of the front end which is only accessible on my Docker network so that Squidex can send me the webhook but my visitors can’t hit it.

All of this code should set up the sync function.

let config = {
    //set up client
    squidexAppName: '',
    squidexClientId: '',
    squidexClientSecret: '',
    squidexEnvironment: '',
    //...
}

export const enum SCHEMAS {
    BRANDS = 'brands',
    PAGES = 'pages',
    MARKETPLACES = 'marketplaces',
    PRODUCT_CATEGORIES = 'product-categories',
    PRODUCTS = 'products',
    SELLERS = 'sellers',
    SLUGS = 'slugs',
    //...
};

export enum SupportedLocales {
    'en-US' = 'en-US',
    'es-US' = 'es-US',
    'fr-CA' = 'fr-CA',
    //...
};

export interface Multilingual<T> {
    [key: string]: T,
    'en-US': T,
    'es-US': T,
    'fr-CA': T,
    //...
};

export interface NonMultilingual<T> {
    [key: string]: T,
    iv: T,
}

export interface ContentDto<T> extends Squidex.ContentDto {
    data?: T;
}

export interface ContentsDto<T> extends Squidex.ContentsDto {
    /** The generic content items. */
    items: ContentDto<T>[];
}

export interface Brand {
    slug: Multilingual<string>,
    //...
}

export interface Marketplace {
    slug: Multilingual<string>,
    //...
}

export interface Page {
    slug: Multilingual<string>,
    //...
}

export interface Product {
    slug: Multilingual<string>,
    //...
}

export interface ProductCategory {
    slug: Multilingual<string>,
    //...
}

export interface Seller {
    slug: Multilingual<string>,
    //...
}

/** This is our cache collection. **/
export interface Slug {
    locale: NonMultilingual<SupportedLocales|string>,
    localizedSlug: NonMultilingual<string>,
    referenceSchema: NonMultilingual<SCHEMAS|string[]>,
    reference: NonMultilingual<string[]>,
}

export const TIMEOUT_IN_SECONDS = 10;

export const client = new SquidexClient({
    appName: config.squidexAppName!,
    clientId: config.squidexClientId!,
    clientSecret: config.squidexClientSecret!,
    environment: config.squidexEnvironment!,
    tokenStore: new SquidexClient.InMemoryTokenStore(),
    // tokenStore: new SquidexStorageTokenStore() // Keep the tokens in the local store.
    // tokenStore: new SquidexStorageTokenStore(sessionStorage, "CustomKey")
});

export const getContents = async <T>(schema: SCHEMAS|string) => (
    await client.contents.getContents(schema, { }, { timeoutInSeconds: TIMEOUT_IN_SECONDS })
) as ContentsDto<T>;

export const getContentsUsingJsonQuery = async <T>(schema: SCHEMAS|string, jsonQuery: string|undefined = undefined) => (
    await client.contents.getContents(schema, { q: jsonQuery }, { timeoutInSeconds: TIMEOUT_IN_SECONDS })
) as ContentsDto<T>;

export const getAllSlugs = async () =>
    await getContents<Slug>(SCHEMAS.SLUGS);

Then this is what I did to sync it. Feel free to soak this method into your thoughts, because for me it is doing exactly what I need it to. If you wanted to set this up more generally you would extrapolate the process and restrict it to a single projection. But since I needed to roll my own, I set the whole thing up as a batch operation.

export async function performSyncLocalizedSlugs() {
    console.log("Begin sync localized slugs.")
    let allSlugs = await getAllSlugs();
    let allPages = await getContentsUsingJsonQuery<Page>(SCHEMAS.PAGES);
    let allBrands = await getContentsUsingJsonQuery<Brand>(SCHEMAS.BRANDS);
    let allProducts = await getContentsUsingJsonQuery<Product>(SCHEMAS.PRODUCTS);
    let allProductCategories = await getContentsUsingJsonQuery<ProductCategory>(SCHEMAS.PRODUCT_CATEGORIES);
    let allSellers = await getContentsUsingJsonQuery<Seller>(SCHEMAS.SELLERS);
    const locales = Object.values(SupportedLocales);
    const findSlugInMultilingual = function<T>(slug: Slug, schema: SCHEMAS|string, contents: ContentsDto<T>) {
        for (let i = 0; i < contents.items.length; i++) {
            let item = contents.items[i];
            for (let l = 0; l < locales.length; l++) {
                let locale = locales[l];
                let testSlug = (item.data! as any).slug[locale]
                if (testSlug) {
                    if (slug.locale.iv === locale
                        && slug.localizedSlug.iv === testSlug
                        && slug.referenceSchema.iv === schema
                        && slug.reference.iv.length === 1
                        && slug.reference.iv[0] === item.id) {
                            return item;
                        }
                }
            }
        }
    }
    const findSlugInSlugs = function(locale: SupportedLocales|string, slug: Multilingual<string>, schema: SCHEMAS|string, referenceId: string) {
        for (let i = 0; i < allSlugs.items.length; i++) {
            let testSlug = allSlugs.items[i].data!;
            if (testSlug.localizedSlug.iv === slug[locale]
                && testSlug.locale.iv === locale
                && testSlug.referenceSchema.iv === schema
                && testSlug.reference.iv.length === 1
                && testSlug.reference.iv[0] === referenceId) {
                    return allSlugs.items[i];
            }
        }
    }
    let batchAddSlugsQueue: Slug[] = [];
    allPages.items.forEach((page) => {
        for (let l = 0; l < locales.length; l++) {
            let locale = locales[l];
            let foundSlugDto = findSlugInSlugs(locale, (page.data! as Page).slug, SCHEMAS.PAGES, page.id);
            if (!foundSlugDto) {
                //cache slug for page
                batchAddSlugsQueue.push({
                    locale: { iv: locale },
                    localizedSlug: { iv: (page.data! as Page).slug[locale] },
                    referenceSchema: { iv: SCHEMAS.PAGES },
                    reference: { iv: [page.id] }
                });
            }
        }
    });
    allBrands.items.forEach((brand) => {
        for (let l = 0; l < locales.length; l++) {
            let locale = locales[l];
            let foundSlugDto = findSlugInSlugs(locale, (brand.data! as Brand).slug, SCHEMAS.BRANDS, brand.id);
            if (!foundSlugDto) {
                //cache slug for brand
                batchAddSlugsQueue.push({
                    locale: { iv: locale },
                    localizedSlug: { iv: (brand.data! as Brand).slug[locale] },
                    referenceSchema: { iv: SCHEMAS.BRANDS },
                    reference: { iv: [brand.id] }
                });
            }
        }
    });
    allProducts.items.forEach((product) => {
        for (let l = 0; l < locales.length; l++) {
            let locale = locales[l];
            let foundSlugDto = findSlugInSlugs(locale, (product.data! as Product).slug, SCHEMAS.PRODUCTS, product.id);
            if (!foundSlugDto) {
                //cache slug for product
                batchAddSlugsQueue.push({
                    locale: { iv: locale },
                    localizedSlug: { iv: (product.data! as Product).slug[locale] },
                    referenceSchema: { iv: SCHEMAS.PRODUCTS },
                    reference: { iv: [product.id] }
                });
            }
        }
    });
    allProductCategories.items.forEach((productCategory) => {
        for (let l = 0; l < locales.length; l++) {
            let locale = locales[l];
            let foundSlugDto = findSlugInSlugs(locale, (productCategory.data! as ProductCategory).slug, SCHEMAS.PRODUCT_CATEGORIES, productCategory.id);
            if (!foundSlugDto) {
                //cache slug for product category
                batchAddSlugsQueue.push({
                    locale: { iv: locale },
                    localizedSlug: { iv: (productCategory.data! as ProductCategory).slug[locale] },
                    referenceSchema: { iv: SCHEMAS.PRODUCT_CATEGORIES },
                    reference: { iv: [productCategory.id] }
                });
            }
        }
    });
    let batchRemoveSlugsQueue: string[] = [];
    allSlugs.items.forEach((slugDto) => {
        const doesSlugExistInPages = findSlugInMultilingual<Page>(slugDto.data! as Slug, SCHEMAS.PAGES, allPages);
        const doesSlugExistInBrands = findSlugInMultilingual<Brand>(slugDto.data! as Slug, SCHEMAS.BRANDS, allBrands);
        const doesSlugExistInProducts = findSlugInMultilingual<Product>(slugDto.data! as Slug, SCHEMAS.PRODUCTS, allProducts);
        const doesSlugExistInProductCategories = findSlugInMultilingual<ProductCategory>(slugDto.data! as Slug, SCHEMAS.PRODUCT_CATEGORIES, allProductCategories);
        const doesSlugExistInSellers = findSlugInMultilingual<Seller>(slugDto.data! as Slug, SCHEMAS.SELLERS, allSellers);
        const doesSlugExistElsewhere = doesSlugExistInPages||doesSlugExistInBrands||doesSlugExistInProducts||doesSlugExistInProductCategories||doesSlugExistInSellers;
        const shouldPruneOrphanSlug = !doesSlugExistElsewhere;
        if (shouldPruneOrphanSlug) {
            //prune orphan slugs from cache
            batchRemoveSlugsQueue.push(slugDto.id);
        }
    });
    const MAX_TIME_TO_POST_SLUGS = 60;//s
    console.log("Add", batchAddSlugsQueue.length, "slugs");
    console.log("Remove by id", batchRemoveSlugsQueue.length, "slugs");
    let bulkAddResult = await core.client.contents.postContents(SCHEMAS.SLUGS, { datas: batchAddSlugsQueue as unknown as ContentData[], publish: true }, { timeoutInSeconds: MAX_TIME_TO_POST_SLUGS });
    batchRemoveSlugsQueue.forEach(async (removeId) => {
        await client.contents.deleteContent(SCHEMAS.SLUGS, removeId)
    })
    console.log("End sync localized slugs.")
}

performSyncLocalizedSlugs();

[Sebastian]

I have not read the details of the code, but there is a special endpoint for that:

https://cloud.squidex.io/api/docs#tag/Contents/operation/Contents_BulkUpdateContents

This endpoint requests a list of jobs. Each job can have a query. You can use this query together with an upsert type to implement an create or update functionality, e.g. a request like that:

{
  "jobs": [{
    "query": {
      "filter": {
		"path": "data.slug.iv",
		"op": "eq",
		"value": "my-slug"
	  }
	},
	"type": "Upsert",
	"data": {
	  "..."
	}
  }]
}