Bad Data from Squidex to Algolia

Alexander_Alemayhu · October 14, 2019, 12:06pm

Hei,

I have a big model which sends custom data to Algolia and in many cases the value stored in Algolia ends up being "Invalid JSON". Is there any way to debug this kind of issue? I have tried reducing the amount of data sent but still seeing Invalid JSON for payloads which should be fine.

Screenshot in Algolia

Sebastian · October 14, 2019, 12:17pm

I guess there is a an encoding problem when a string has an invalid character for JSON. I could add an validation step and we might need a helper method to encode this.

Alexander_Alemayhu · October 14, 2019, 12:18pm

Hmm, the model sometimes has strings that have Norwegian text and some urls do have extra backward slash \ due to legacy API crap. Could those be causing the issue?

Sebastian · October 14, 2019, 12:21pm

Very likely. The \ is a control character.

Alexander_Alemayhu · October 14, 2019, 2:13pm

Ok, when can you deploy the helper method to encode this?

Sebastian · October 14, 2019, 2:52pm

I don’t know yet. Are you running on Cloud or OnPremise?

Sebastian · October 14, 2019, 3:17pm

Hi, I just found out, that it is already solved:

github.com

Squidex/squidex/blob/master/tests/Squidex.Domain.Apps.Core.Tests/Operations/HandleRules/RuleEventFormatterTests.cs#L370


public void Should_format_actor(string script)
{
    var @event = new EnrichedContentEvent { Actor = new RefToken(RefTokenType.Client, "android") };


    var result = sut.Format(script, @event);


    Assert.Equal("From client:android", result);
}


[Fact]
public void Should_format_json()
{
    var @event = new EnrichedContentEvent { Actor = new RefToken(RefTokenType.Client, "android") };


    var result = sut.Format("Script(JSON.stringify({ actor: event.actor.toString() }))", @event);


    Assert.Equal("{\"actor\":\"client:android\"}", result);
}


[Fact]
public void Should_format_json_with_special_characters()

Alexander_Alemayhu · October 15, 2019, 3:55am

Running it ourselves and the version is 3.4.0.0.

Sebastian · October 15, 2019, 5:29am

I just asked because of the deployment. But I think the above approach should work.

Alexander_Alemayhu · October 15, 2019, 11:04am

I tried pulling in the latest Squidex release via Docker but still seeing invalid JSON. Would love to get this resolved due to pressure to wrap up a project.

Thanks.

Sebastian · October 15, 2019, 11:11am

Have you tested my solution?

Alexander_Alemayhu · October 15, 2019, 11:17am

No, don’t think so since the latest docker image seems to be 6 days ago. Could you deploy a new version or should I use squidex/squidex/dev-3171?

Sebastian · October 15, 2019, 11:28am

I have not changed anything, the JSON.stringify is there since a year or so

Alexander_Alemayhu · October 15, 2019, 11:39am

Ah, then do you have any feedback on how we can debug the rules resulting in the invalid JSON?

Sebastian · October 15, 2019, 11:53am

You are right, i have forgotten this. I can probably implement it tomorrow. Today is a horrible day, meetings over meetings.

Alexander_Alemayhu · October 15, 2019, 11:56am

Cool, will check again tomorrow.

Have a nice day!

Thanks.

Sebastian · October 16, 2019, 3:30pm

I have made a small change to also include the exception message to the error.

Alexander_Alemayhu · October 17, 2019, 12:13pm

Could you create a release tag so we can deploy it? I tried using the build image but that just breaks stuff

Uncaught ReferenceError: ngDevMode is not defined
    at Module.<anonymous> (app.js?ef8b3c2bdc7d8f9e8b65:2)
    at Module.<anonymous> (app.js?ef8b3c2bdc7d8f9e8b65:2)
    at i (app.js?ef8b3c2bdc7d8f9e8b65:2)
    at Module.<anonymous> (app.js?ef8b3c2bdc7d8f9e8b65:2)
    at i (app.js?ef8b3c2bdc7d8f9e8b65:2)
    at app.js?ef8b3c2bdc7d8f9e8b65:2
    at app.js?ef8b3c2bdc7d8f9e8b65:2

Sebastian · October 17, 2019, 2:27pm

The build tag has the same build process like the release tags, so I have to figure out where this is common from first.

Sebastian · October 17, 2019, 7:41pm

Hi, I have found the problem. This fix should solve it: https://build.squidex.io/Squidex/squidex/3195

Seems to be an angular error: https://github.com/angular/angular/issues/31595