Tune the Search Strategy

To make a basic search query, simply pass a question in the query parameter of the /find or /ask endpoint. This will return the most relevant data for your query based on Nuclia's semantic and keyword search capabilities.

To improve the accuracy of your search results, consider using the following parameters and strategies.

Search Modes

Nuclia offers several search modes: semantic search, keyword search, fulltext search, and graph search:

• Semantic search is based on the meaning of the query and the content of the paragraphs, it will retrieve the paragraphs that are semantically close to the query. It is the most powerful search mode.
• Keyword search is based on the keywords of the query and the content of the paragraphs, it will retrieve the paragraphs that contain the keywords of the query. It is a more naive search mode but it can be very relevant when searching for a specific term (typically a brand name, a product name, etc.).
• Fulltext search is based on the keywords of the query and the content of the resources, it will retrieve the resources that contain the query's terms. It only applies to /find endpoint.
• Graph search is based on the entities of the query and the content of the resources, it will retrieve the resources that contain the entities of the query. It is a very powerful search mode when the query is about a specific entity (like a person, a location, an organization, etc.).

You can apply multiple search modes to a single query. The /ask endpoint uses semantic search, keyword search, and graph search by default. The /find endpoint uses semantic search and fulltext search by default.

But you might want to change the default behavior. To do so, you can use the features parameter.

Typically, when searching with a different language than the resource's language, keyword search might be inconsistent, matching words that are written the same but have different meanings in the two languages. In this case, you can use semantic search only by passing:

features: ['semantic']

Rephrase the query

The rephrase parameter allows you to optimize the query for semantic search.

Typically, when the user's query is a set of keywords (like "prune apple tree period"), it might perform badly on semantic search. To avoid this, you can pass rephrase=true so the query is rephrased to a more natural language question (like "When is the best time to prune an apple tree?").

The generative model used for rephrase can be chosen using the generative_model parameter in the /find or /ask request.

Auto-filter

When passing autofilter=true, the user's query will be scanned for keywords that are likely to be named entities (or NER) and they will be automatically used as filters. This is useful when the user's query is a question like "How to replace the brakes on the XMB55 bike?", because this query might have very good semantic matches on any resource explaining how to replace brakes on a bike, but the user is only interested in the XMB55 bike.

Filters

See Search filters for more information on how to use filters.

Filter by security groups

If you need to restrict the search depending on the user's security groups, you can use the security.groups parameter.

See Restrict access to resources for more information.

Minimum score

All search endpoints provide parameters to filter out results that are not good enough. For instance, the POST ask, find and search provide the min_score parameter on the payload with which you can control:

• Semantic score: the measure of meaning or semantic similarity between the search vector and the results. When not provided, NucliaDB will use the minimum score associated to the semantic model configured for the Knowledge Box. Read more on the subsection below.

• BM25 score: BM25 is the ranking function used by NucliaDB text search index to rank matching documents according to their relevance to a given search query. NucliaDB does not filter by BM25 score by default.

An example payload would be:

{
  "query": "Hakuna Matata",
  "min_score": {
    "bm25": 4.0,
    "semantic": 1.2
  }
}

On the GET version of the search and find endpoints, these can be specified as query params:

/api/v1/kb/kbid/search?query=Hakuna%20Matata&min_score_bm25=4.0&min_score_semantic=1.2

BM25 scores

Theoretically, the range of the BM25 score can be anything from 0 to infinity. In practice, however, scores are typically within a certain range (e.g., 0 to 10). The score depends on several factors, including the frequency of the term in the document, the length of the document, the average length of documents in the collection, and the frequency of the term in the entire document collection.

A higher BM25 score indicates a higher relevance of the document to the search query. However, because the score is not normalized, the absolute value of the score is not directly interpretable and is not comparable across different queries or document collections.

A score of 0 indicates that the document has no relevance to the query (i.e., none of the query terms appear in the document).

Semantic scores

The range of the semantic similarity is different depending on the semantic model in use and their associated similarity function. At the time of writing, there are two different similarity functions used by NucliaDB's vectors index:

• Dot product: also known as scalar product. The range is any real number. The dot product of two vectors will be higher the more similar the vectors are.

• Cosine: the range of the cosine similarity function is between -1 and 1. However, in practice it is typically between 0 and 1 because negative scores indicate that vectors are not similar. A score of 1 means that the two vectors are identical.

As mentioned above, each semantic model uses a different similarity function. Moreover, Nuclia has pre-defined a min score for each model to provide a good generic search experience. Below is a list with the semantic models supported at the time of writing with the associated similarity function and the default min score used by NucliaDB's search engine:

Semantic Model	Similarity function	Score range	Default min score
en (English)	Cosine	real numbers from -1 to 1	0.7
multilingual-2023-02-21	Dot product	any real number	1.5
multilingual-2023-08-16	Dot product	any real number	0.7

note

For more specific search use-cases, it is recommended that you experiment different min scores on your dataset and the types of queries that are expected. To that effect, you can use the parameters explained above.

Facets

By using the faceted parameter, you will get a facets attribute in paragraphs, sentences and fulltext.

This parameter takes on the same values as the filters parameter.

Examples:

• To get the total amount of matches for each image file type (like jpg, png, gif, etc.), use:

  faceted=/icon/image

• To get the total amount of matches for each language (like en, it, fr, etc.), use:

  faceted=/metadata.language

Highlight matching words

By setting the split parameter to true, you will get the start and end positions of each matching word in text blocks and fulltext results.

If you additionally set the highlight parameter to true, the matching words are enclosed into <mark> tags.