[SOLVED] ElasticSearch not work in 5.0.0

Jonas_Tang · October 15, 2020, 6:38pm

The full text search by ElasticSearch can not work, it always get a 500 error in the admin UI.
And a http status 400 error in the log:

{
  "logLevel": "Error",
  "message": "An unexpected exception has occurred.",
  "app": {
    "name": "Squidex",
    "version": "1.0.0.0",
    "sessionId": "3f8d7146-53ec-4153-ac82-8f605fbd73db"
  },
  "web": {
    "requestId": "|98c412cf-4bfb8a86c1dc6603.",
    "requestPath": "/content/db/artist",
    "requestMethod": "GET",
    "routeValues": {
      "area": "Api",
      "action": "GetContents",
      "controller": "Contents"
    }
  },
  "timestamp": "2020-10-15T18:19:50Z",
  "exception": {
    "type": "Elasticsearch.Net.ElasticsearchClientException",
    "message": "Request failed to execute. Call: Status code 400 from: POST /squidex/_search. ServerError: Type: x_content_parse_exception Reason: \u0022[1:54] [bool] failed to parse field [must]\u0022 CausedBy: \u0022Type: illegal_argument_exception Reason: \u0022value cannot be null\u0022\u0022",
    "stackTrace": "   at Squidex.Domain.Apps.Entities.Contents.Text.Elastic.ElasticSearchTextIndex.SearchAsync(String queryText, IAppEntity app, SearchFilter filter, SearchScope scope) in /Users/jonas/Desktop/projects/pacc/squidex/backend/src/Squidex.Domain.Apps.Entities/Contents/Text/Elastic/ElasticSearchTextIndex.cs:line 194\n   at Squidex.Domain.Apps.Entities.MongoDb.Contents.Operations.QueryContentsByQuery.DoAsync(IAppEntity app, ISchemaEntity schema, ClrQuery query, SearchScope scope) in /Users/jonas/Desktop/projects/pacc/squidex/backend/src/Squidex.Domain.Apps.Entities.MongoDb/Contents/Operations/QueryContentsByQuery.cs:line 86\n   at Squidex.Domain.Apps.Entities.MongoDb.Contents.MongoContentCollectionAll.QueryAsync(IAppEntity app, ISchemaEntity schema, ClrQuery query) in /Users/jonas/Desktop/projects/pacc/squidex/backend/src/Squidex.Domain.Apps.Entities.MongoDb/Contents/MongoContentCollectionAll.cs:line 71\n   at Squidex.Domain.Apps.Entities.Contents.Queries.ContentQueryService.QueryByQueryAsync(Context context, ISchemaEntity schema, Q query) in /Users/jonas/Desktop/projects/pacc/squidex/backend/src/Squidex.Domain.Apps.Entities/Contents/Queries/ContentQueryService.cs:line 216\n   at Squidex.Domain.Apps.Entities.Contents.Queries.ContentQueryService.QueryAsync(Context context, String schemaIdOrName, Q query) in /Users/jonas/Desktop/projects/pacc/squidex/backend/src/Squidex.Domain.Apps.Entities/Contents/Queries/ContentQueryService.cs:line 102\n   at Squidex.Areas.Api.Controllers.Contents.ContentsController.GetContents(String app, String name, String ids, String q) in /Users/jonas/Desktop/projects/pacc/squidex/backend/src/Squidex/Areas/Api/Controllers/Contents/ContentsController.cs:line 218\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeActionMethodAsync\u003Eg__Awaited|12_0(ControllerActionInvoker invoker, ValueTask\u00601 actionResultValueTask)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeNextActionFilterAsync\u003Eg__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State\u0026 next, Scope\u0026 scope, Object\u0026 state, Boolean\u0026 isCompleted)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.\u003CInvokeInnerFilterAsync\u003Eg__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.\u003CInvokeNextExceptionFilterAsync\u003Eg__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)"
  }
}

I checked the ElasticSearch indices, It creates indices when create an item, update and delete won’t update.

The index in ElasticSearch looks like below:
the appId, contentId, schemaId are empty, i think that’s why we get the 400 error from:

https://github.com/Squidex/squidex/blob/master/backend/src/Squidex.Domain.Apps.Entities/Contents/Text/Elastic/ElasticSearchTextIndex.cs#L135

{
	"_index": "paccdb",
	"_type": "_doc",
	"_id": "6d9826e9-9342-4f16-baf7-007d36a48112--c4e15469-b392-4cfd-9f25-db43f1691fc5_1",
	"_version": 1,
	"_seq_no": 1,
	"_primary_term": 1,
	"found": true,
	"_source": {
		"appId": {},
		"appName": "db",
		"contentId": {},
		"schemaId": {},
		"schemaName": "artist",
		"serveAll": true,
		"servePublished": true,
		"texts": {
			"iv": "彭俊发 男 1964-07-01T00:00:00Z 汉 上海市 上海市 奉贤区 66b937f8-7317-4c3f-9b29-9a44b2bde3eb 传统木作技艺 区（市）级传承人 测试 eae4a32a-c151-4af4-aedd-e18d897cf3fa bb6feb41-f974-457b-b277-1ec07dd4cea0 da0f3db6-29fa-4340-ab0b-15593281bb7f 1183e25b-d103-4a4e-a107-eec8d32b9e76 863a32ce-591c-4513-8504-243b9907326b b53fbab4-f0a2-4a60-ad2c-d53e9f46a779 da89a893-1ff3-420d-accc-1c7dfebb142d e5528e3a-2ee6-4576-ac5c-07160b9e55e8 97897ee9-8ae0-40fe-a6e8-fcae1c0c379c 2f0c4e23-fbca-4e77-9fd3-0090a864ac86 03084a09-6544-4502-9c2a-2a1941b8b17e a087c37c-49c5-498d-95c2-534c5a0d7216"
		}
	}
}

By the way, can we resolve the reference id to related data, the content id in the texts is meaningless.

[ x] Checked the logs and have uploaded a log file and provided a link because I found something suspicious there. Please do not post the log file in the topic because very often something important is missing.

I’m submitting a…

[ ] Regression (a behavior that stopped working in a new release)
[ x] Bug report
[ ] Performance issue
[ x] Documentation issue or request

Current behavior

Expected behavior

Minimal reproduction of the problem

Environment

[ ] Self hosted with docker
[ ] Self hosted with IIS
[ x] Self hosted with other version
[ ] Cloud version

Version: [VERSION]

Browser:

[ x] Chrome (desktop)
[ ] Chrome (Android)
[ ] Chrome (iOS)
[ ] Firefox
[ ] Safari (desktop)
[ ] Safari (iOS)
[ ] IE
[ ] Edge

Others:

Sebastian · October 15, 2020, 8:26pm

Thats an issue with 5.0. I have pushed an potential fix, but I have no time today to test it carefully.

Elastic Search is experimental anyway.

Jonas_Tang · October 16, 2020, 3:08am

Thanks for you quick reply.
ES is very important for the languages like Chinese, Japanese etc, since these languages are not supported by mongodb text search, see https://docs.mongodb.com/manual/reference/text-search-languages/

Sebastian · October 16, 2020, 5:49am

For MongoDB I use only the simple tokenizer anyway. It just means that synonyms and stop words are not supported my MongoDB but you can still use the search.

Jonas_Tang · October 16, 2020, 9:26am

I see, it works in languages which have whitespace tokenizer (e.g. English/German) by this way. But Chinese/Japanese don’t have.
If we have a sentence in the database like “可以将正向最大匹配方法和逆向最大匹配方法结合起来构成双向匹配法”, there are lots of word in this text, but if you want to get a matched result, you have to input the whole text.

Sebastian · October 16, 2020, 10:15am

I see. I have tested it with Hebrew, but I think the use normal whitespaces as well. I had to do a lot with CJK languages when I was writing a word count calculator.

If you want you can also provide a PR for algolia integration, could be work as well for your use case.

Jonas_Tang · October 16, 2020, 4:32pm

Thanks for your update, but it seems still not working, another error occured:

{
  "logLevel": "Fatal",
  "action": "OnEventsAsync",
  "status": "Failed",
  "eventConsumer": "TextIndexer5",
  "app": {
    "name": "Squidex",
    "version": "1.0.0.0",
    "sessionId": "ec1da379-b4a5-4be6-bcfe-b683a9e3949b"
  },
  "timestamp": "2020-10-16T16:30:12Z",
  "exception": {
    "type": "System.InvalidOperationException",
    "message": "Failed with ${\u0022error\u0022:{\u0022root_cause\u0022:[{\u0022type\u0022:\u0022mapper_parsing_exception\u0022,\u0022reason\u0022:\u0022object mapping for [appId] tried to parse field [appId] as object, but found a concrete value\u0022}],\u0022type\u0022:\u0022mapper_parsing_exception\u0022,\u0022reason\u0022:\u0022object mapping for [appId] tried to parse field [appId] as object, but found a concrete value\u0022},\u0022status\u0022:400}",
    "stackTrace": "   at Squidex.Domain.Apps.Entities.Contents.Text.Elastic.ElasticSearchTextIndex.UpsertAsync(UpsertIndexEntry upsert) in /Users/jonas/Desktop/projects/pacc/squidex/backend/src/Squidex.Domain.Apps.Entities/Contents/Text/Elastic/ElasticSearchTextIndex.cs:line 86\n   at Squidex.Domain.Apps.Entities.Contents.Text.Elastic.ElasticSearchTextIndex.ExecuteAsync(IndexCommand[] commands) in /Users/jonas/Desktop/projects/pacc/squidex/backend/src/Squidex.Domain.Apps.Entities/Contents/Text/Elastic/ElasticSearchTextIndex.cs:line 51\n   at Squidex.Domain.Apps.Entities.Contents.Text.TextIndexingProcess.Updates.WriteAsync(ITextIndex textIndex, ITextIndexerState textIndexerState) in /Users/jonas/Desktop/projects/pacc/squidex/backend/src/Squidex.Domain.Apps.Entities/Contents/Text/TextIndexingProcess.cs:line 65\n   at Squidex.Domain.Apps.Entities.Contents.Text.TextIndexingProcess.On(IEnumerable\u00601 events) in /Users/jonas/Desktop/projects/pacc/squidex/backend/src/Squidex.Domain.Apps.Entities/Contents/Text/TextIndexingProcess.cs:line 350\n   at Squidex.Infrastructure.EventSourcing.Grains.EventConsumerGrain.DispatchAsync(IReadOnlyList\u00601 events) in /Users/jonas/Desktop/projects/pacc/squidex/backend/src/Squidex.Infrastructure/EventSourcing/Grains/EventConsumerGrain.cs:line 185\n   at Squidex.Infrastructure.EventSourcing.Grains.EventConsumerGrain.\u003C\u003Ec__DisplayClass16_0.\u003C\u003COnEventsAsync\u003Eb__0\u003Ed.MoveNext() in /Users/jonas/Desktop/projects/pacc/squidex/backend/src/Squidex.Infrastructure/EventSourcing/Grains/EventConsumerGrain.cs:line 93\n--- End of stack trace from previous location where exception was thrown ---\n   at Squidex.Infrastructure.EventSourcing.Grains.EventConsumerGrain.DoAndUpdateStateAsync(Func\u00601 action, String caller) in /Users/jonas/Desktop/projects/pacc/squidex/backend/src/Squidex.Infrastructure/EventSourcing/Grains/EventConsumerGrain.cs:line 205"
  }
}

Sebastian · October 16, 2020, 6:37pm

The current build should fix it, I have done a lot of testing.

But CJK was not working out of the box. I had to configure all analyzers properly:

github.com

Squidex/squidex/blob/master/backend/src/Squidex.Domain.Apps.Entities/Contents/Text/Elastic/ElasticSearchMapping.cs

// ==========================================================================
//  Squidex Headless CMS
// ==========================================================================
//  Copyright (c) Squidex UG (haftungsbeschraenkt)
//  All rights reserved. Licensed under the MIT license.
// ==========================================================================

using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using Elasticsearch.Net;

namespace Squidex.Domain.Apps.Entities.Contents.Text.Elastic
{
    public static class ElasticSearchMapping
    {
        public static async Task ApplyAsync(IElasticLowLevelClient elastic, string indexName, CancellationToken ct = default)
        {
            var query = new

This file has been truncated. show original

I have also done 2 more improvements to elastic search implementation:

Turn on fuzzy search with a leading “~”, e.g. “wold~” for “world”.
Search only in one language with “en:World”

Sebastian · October 24, 2020, 9:46am