[UNLIKELY] Content Fulltext Search Improvement

NeoMoritsiqu · September 19, 2021, 10:37am

Hi Sebastian,

I have idea about development that can search fulltext on all content. Such an improvement for CMS and Web sites would be very nice. This will not be an internal search, it will be an improvement that can give out search results.

Currently it is very difficult to develop on squidex for CMS and Web site searches.

Some settings can be adjusted in the properties section of the schemes.

This service must be located on the Content API.

The output of the service should look like this:

Search keyword: “sample”

[
  {
    "schemaId": "c64f56c3-458f-482b-bc5c-a292caa6cc94",
    "schema": "asd",
    "contentId": "ae5d0bce-a66d-4d87-b3e6-e83db66f72cb",
    "title": null,
    "url": null,
    "found": [
      {
        "field": "examplefield",
        "content": "this is a sample content"
      }
    ]
  },
  {
    "schemaId": "b34c0929-c24f-49b0-82e7-0b8b48eeac3d",
    "schema": "test",
    "contentId": "9076bb7b-9889-4ecb-8ee3-b2247ff43753",
    "title": "Test content title",
    "url": "test-content-title",
    "found": [
      {
        "field": "samplefield",
        "content": "hello world this is sample text"
      },
      {
        "field": "samplefield2",
        "content": "this is a sample text"
      }
    ]
  }
]

Of course, this idea can be improved. I’m curious about your comments.

Thank you.

Sebastian · September 19, 2021, 1:00pm

Hi,
thanks for your mock-ups. Well done.

Nevertheless I think it is not the best idea.

Current State

Let my explain the current state in Squidex first. At the moment Squidex has a full text implementation. The implementation is very basic and has been migrated from a Lucene backed implementation to a implementation using MongoDb or ElasticSearch. At the moment all text fields are added to an index and the ElasticSearch implementation also considers the different languages for localized fields.

The Lucene implementation produced better results than the MongoDB implementation but it was buggy and hard to maintain. Therefore the implementation has been shifted.

What is the problem?

I am not sure how familiar you are with full text indexes, but the implementation works most of the time with a so called reverse index. Lets consider you have two documents.

| Id | Texts                   |
| 1  | Hello User              | 
| 2  | Hello, how is it going? |

The full text index splits the text into its words (or word stems). Then it associated the ids to the words. Because some words like and and used so often these words are added to a blacklist, so called stopwords and no added to the index.

So after this process is done you get the following results

| Word   | Ids  |
| Hello  | 1, 2 |
| User   | 1    |
| Go     | 3    |

In this case how, is and it are treated as stopwords and going is reduced to its stem go.

Problem 1: A lot to configure

To make the full text work the best way a lot of settings have to be made:

Stemming and word splitting (tokenization) differes between the languages, so per field you have to decide which language should be used.
Stop words also differ per language and you have to define per field which stopwords should be used. But sometimes you also want to have custom stopwords. For example if you have a travel port you wanna exclude words like hotel and flight from your results, because they are basically part of every content item and it would produce the wrong results.
Some fields are more important than others, for example the title is more important than then the text, so you wanna add a weight to some fields.
Some fields contain words that are hard to understand for the search engine. For example the IATA code for airports. You wanna treat these fields differently.
You also wanna configure synonyms, so that the word does not have to match and approximity search so that typos are acceptable.

There are more settings and I am not an expert for full text search, but you can have a look at the documentation of Algolia or Elastic Search to understand all the different options.

Without deep understanding of these parameters you will not get good results. There is no automatic solution that does everything for you.

So when we really wanna integrate full text search into Squidex we have to make these parameters and configuration options available to the developer.

But we have to consider the different implementations and have to study what the different solutions offer. Just a list of servers and technologies:

Lucene based (a Java full text search engine)
- Lucene itself
- Elastic Search
- Solr
Algolia (SaaS)
Databases (most databases have basic full text support today)
MeiliSearch

Then we have 2 options:

Support all parameters that are available by at least one of these engines and make clear why this parameter cannot be used in the current installation.
Support only common parameters.

We also have to implement processes like reindexing because it might be necessary to index all your content items again after you made a change to the stopwords.

Problem 2: Document transformation

The content items itself are not the best representation for a full text index. Lets come back to our original sample with the travel portal. A travel offer has a reference to the airport and the destination and the hotel. In the content these references are only represented as IDs and provide no value for the search engine. So we must also configure which fields of referenced items should be added to the document for the full text search engine.

After all it is a lot of effort to bring a very good fulltext solution and I am not sure if it would provide enough value.

NeoMoritsiqu · September 19, 2021, 1:08pm

Thank you for the in-depth explanation. I got a much better idea with the explanations. So how can we produce a similar solution? How can we produce a good search solution for websites. How can we improve this solution? I think it’s a topic worth thinking about. There is no good solution for this at the moment.

Thank you for all your support.

Sebastian · September 19, 2021, 1:53pm

You can use rules to publish your changes to Algolia or Elastic and then query your content from there.

OR

We could make a simple integration in addition to that so that Algolia gets integrated into Squidex. So basically you would use Rules to publish to Algolia and the search engine integration to trigger the search only.

NeoMoritsiqu · September 24, 2021, 9:17am

So what if we develop this kind of development in a way that we can search on the contentsRepo, not as a fulltext search?

Sebastian · September 24, 2021, 5:05pm

You mean a “contains” search? You can do that with filter, but it is very slow, because it is always a table scan.

Vishnuvardhan-Dudda · July 25, 2023, 10:10am

Hi Sebastian,

Any better approach available now for full text search on Squidex schema.
I am searching over the Squidex portal as well as on the browser to know.

Could you please help us to have some knowledge on Squidex schema with full text search.

Thanks & Regards,
Vishnu D

Sebastian · July 25, 2023, 10:28am

In my previous long answer I have made it clear that it is somehow impossible to provide a state of the art search solution and this has not been changed yet. If you disagree or if you have any insights from other products, I am very happy to discuss that.

For self hosting you can also connect Squidex with elastic search and it could provide better default results.

Vishnuvardhan-Dudda · July 25, 2023, 12:33pm

Hi Sebastian,
Thanks for your quick reply, Do not think wrong asking same query.

Below provided/attached screenshot is the schema details which i would like to search data on any column/field(full text search). application is not docker imaged.

Please suggest us if any help from you.

Regards,
Vishnu D

Sebastian · July 25, 2023, 12:43pm

What exactly is your question?

Vishnuvardhan-Dudda · July 25, 2023, 1:20pm

Sebastian,

I am sorry to disturbing your time. Please forgive me if i am confusing you.

I would like to implement full text search(all fields) on earlier attached schema. we are using Squidex database in our Dot Net Core application.

Still any more information required please let me know.

Regards,
Vishnu D

slalFe · July 25, 2023, 1:54pm

All your fields appear to be string fields so they should already automatically be included in the Squidex built-in full text search:
https://docs.squidex.io/02-documentation/developer-guides/api-overview/api#full-text-searches

Sebastian · July 25, 2023, 7:47pm

I think your question has very little to do with this topic. Perhaps this helps: https://docs.squidex.io/02-documentation/developer-guides/api-overview/api

Sebastian · July 25, 2023, 7:47pm