How to query a Knowledge Box

Nuclia supports 3 different search endpoints:

/search: returns several result sets according the different search techniques (full-text, fuzzy, semantic).
/find: returns a single result set where all different results are merged into a hierarchical structure.
/chat: streams a generative answer and a single result set where all different results are merged into a hierarchical structure.

All endpoints support the same query parameters.

Search parameters

Query

Simple query

A simple text search can be performed using a plain text value.

Example:

This query would return results containing the words little and prince:

Little Prince

By putting words into quotes you can search for an exact match:

"Little Prince"

In this case, you will only get results containing the word sequence little prince.

Using the minus sign - in front of a word you can exclude a word from the search:

Little Prince -sheep

This query would return results containing the words little and prince but not sheep.

Filters

The filters parameter allows you to filter the results depending on the value of different properties provided on the resource.

The following attributes are supported:

/origin.tags: tags defined in the resource's origin property Example: /origin.tags/blue, /origin.tags/green
/classification.labels: labels: /classification.labels/{labelset}/{label} Example: /classification.labels/movie-genre/science-fiction
/icon: mime type of resource Example: /icon/application/pdf or /icon/movie/mp4
/metadata.status: processing status Example: /metadata.status/PROCESSED, /metadata.status/PENDING or /metadata.status/ERROR
/entities: resource entities: /entities/{entity-type}/{entity-id} Example: /entities/CITY/Barcelona
/metadata.language: primary language of the document Example: /metadata.language/ca for catalan language
/metadata.languages: all other detected languages Example: /metadata.languages/tr for turkish language
/field: keyword fields values /field/fieldname/value Example: /field/countries/Slovenia
/field-values: flattened keyword field values /field-values/{value} Example: /field-values/Slovenia
/origin.metadata: metadata provided by the user Example: /origin.metadata/fieldname/value
origin.path: path of the resource in the source system. It will match any path starting with the provided value. Example: /origin.path/Users/JohnDoe/Documents will match files in the Documents folder of the JohnDoe user, but also the ones in Documents/Work or Documents/Personal, etc.

Examples:

To retrieve PNG images only, use:
```
filters=/icon/image/png
```
To retrieve results in which the principal language is Italian, use:
```
filters=/metadata.language/it
```
To retrieve results referring to the UNESCO organization, use:
```
filters=/entities/ORG/UNESCO
```

Filters can be combined by repeating the filters parameter. This example will retrieve results which are PDF and which are referring to the UNESCO organization:

filters=/icon/application/pdf&filters=/entities/ORG/UNESCO

Advanced filtering

As shown above, combining multiple filters will implicitly return the intersection (i.e: AND operator) between the specified filters. If your use-case needs more complex filtering expressions, you can use the POST versions of the search endpoints to provide a filtering expression. Filtering expressions accept the following keys: all, any, none and not_all. Here are some examples:

all

{
  "filters": [
    {"all": ["/icon/application/pdf", "/entities/ORG/UNESCO"]}
  ]
}

Which would be equivalent to the last example of the previous section: it will return resources that are PDF and have the UNESCO entity associated with them.

any

{
  "filters": [
    {"any": ["/icon/application/pdf", "/icon/movie/mp4"]}
  ]
}

Will return resources that are either PDF or mp4 videos. This is equivalent to the OR logical operation.

none

{
  "filters": [
    {"none": ["/icon/application/pdf", "/icon/movie/mp4"]}
  ]
}

Will return results from documents that are neither PDF nor mp4 videos. This is equivalent to the NOT(a OR b) logical expression.

not_all

{
  "filters": [
    {"not_all": ["/icon/application/pdf", "/entities/ORG/UNESCO"]}
  ]
}

Essentially, it will return the complementary set of results to the all example: all documents except those that are PDFs and also have UNESCO entity related to. This is equivalent to the NOT(a AND b) logical expression.

Combining

If you need even more complex filtering expressions, you can combine multiple expression terms as more elements of the filters list:

{
  "filters": [
    {"all": ["/icon/application/pdf"]},
    {"any": ["/entities/ORG/UNESCO", "/entities/GPE/US"]},
  ]
}

And the returned result will be the implicit intersection (i.e: AND) of all expressions combined. In this example, it will return all documents that are PDF and that have either UNESCO or US as a related entity.

Date filtering

You can filter on the creation date using:

range_creation_start
range_creation_end

Examples:

To get all resources created between 2023-01-01 and 2023-12-31:

range_creation_start=2023-01-01T00:00:00.000Z&range_creation_end=2023-12-31T23:59:59.000Z

To get all resources created after 2023-01-01:
```
range_creation_start=2023-01-01T00:00:00.000Z
```

Filtering will be based on the origin.created value if provided in the resource, otherwise it will default to the resource creation date (created).

note

Please note: all resources created before 2023-11-02 will have to be reprocessed for origin.created to be filterable.

Similarly, you can filter on the modification date using:

range_modification_start
range_modification_end

Search in a specific field

To restrict the search to a specific field you can use the field parameter. It supports different field types:

a: generic fields (= basic attributes, like title or summary)
t: text fields
f: file fields
u: link fields

Example:

fields=a/title

To search in several fields, the parameter can be repeated:

fields=a/title&fields=a/summary

Regarding content fields, when used through the resource /search endpoint it allows you to restrict the search to one piece of content only, and when used through the main /search endpoint it allows you to restrict the search to all content having a given id in all resources.

Minimum score

All search endpoints provide parameters to filter out results that are not good enough. For instance, the POST chat, find and search provide the min_score parameter on the payload with which you can control:

Semantic score: the measure of meaning or semantic similarity between the search vector and the results. When not provided, NucliaDB will use the minimum score associated to the semantic model configured for the Knowledge Box. Read more on the subsection below.
BM25 score: BM25 is the ranking function used by NucliaDB text search index to rank matching documents according to their relevance to a given search query. NucliaDB does not filter by BM25 score by default.

An example payload would be:

{
  "query": "Hakuna Matata",
  "min_score": {
    "bm25": 4.0,
    "semantic": 1.2
  }
}

On the GET version of the search and find endpoints, these can be specified as query params:

/api/v1/kb/kbid/search?query=Hakuna%20Matata&min_score_bm25=4.0&min_score_semantic=1.2

BM25 scores

Theoretically, the range of the BM25 score can be anything from 0 to infinity. In practice, however, scores are typically within a certain range (e.g., 0 to 10). The score depends on several factors, including the frequency of the term in the document, the length of the document, the average length of documents in the collection, and the frequency of the term in the entire document collection.

A higher BM25 score indicates a higher relevance of the document to the search query. However, because the score is not normalized, the absolute value of the score is not directly interpretable and is not comparable across different queries or document collections.

A score of 0 indicates that the document has no relevance to the query (i.e., none of the query terms appear in the document).

Semantic scores

The range of the semantic similarity is different depending on the semantic model in use and their associated similarity function. At the time of writing, there are two different similarity functions used by NucliaDB's vectors index:

Dot product: also known as scalar product. The range is any real number. The dot product of two vectors will be higher the more similar the vectors are.
Cosine: the range of the cosine similarity function is between -1 and 1. However, in practice it is typically between 0 and 1 because negative scores indicate that vectors are not similar. A score of 1 means that the two vectors are identical.

As mentioned above, each semantic model uses a different similarity function. Moreover, Nuclia has pre-defined a min score for each model to provide a good generic search experience. Below is a list with the semantic models supported at the time of writing with the associated similarity function and the default min score used by NucliaDB's search engine:

Semantic Model	Similarity function	Score range	Default min score
en (English)	Cosine	real numbers from -1 to 1	0.7
multilingual-2023-02-21	Dot product	any real number	1.5
multilingual-2023-08-16	Dot product	any real number	0.7

note

For more specific search use-cases, it is recommended that you experiment different min scores on your dataset and the types of queries that are expected. To that effect, you can use the parameters explained above.

Result options

Features

A search query can be executed against different targets. The target is defined by the features parameter which supports 4 values:

document: the query is executed as full-text search against all resource texts (including attributes like title or summary, and all content fields). The result will contain fulltext (listing text sequences matching the query) and resources (listing resources matching the query).
paragraph: the query is executed as fuzzy search against all text block texts. The result will contain paragraphs (listing text blocks matching the query) and resources (listing resources containing these text blocks).
vector: the query is executed as semantic search against all resource texts. The result will contain sentences (listing sentences semantically close to the query) and resources (listing resources containing these sentences).
relations: the query is executed as graph search against all resource entities. The result will contain resources (listing resources related to the entities identified in the query).

These features can be combined by repeating the features parameter:

features=document&features=vector

By using the faceted parameter, you will get a facets attribute in paragraphs, sentences and fulltext.

This parameter takes on the same values as the filters parameter.

Examples:

To get the total amount of matches for each image file type (like jpg, png, gif, etc.), use:
```
faceted=/icon/image
```
To get the total amount of matches for each language (like en, it, fr, etc.), use:
```
faceted=/metadata.language
```

Highlight matching words

By setting the split parameter to true, you will get the start and end positions of each matching word in text blocks and fulltext results.

If you additionally set the highlight parameter to true, the matching words are enclosed into <mark> tags.

How to call the search endpoint

To search in all resources, the search endpoints are:

https://<zone>.nuclia.cloud/api/v1/kb/<your-knowledge-box-id>/search
https://<zone>.nuclia.cloud/api/v1/kb/<your-knowledge-box-id>/find

Search endpoints can be called with a GET or a POST request.

A typical curl command to call the search endpoint is:

https://<zone>.nuclia.cloud/api/v1/kb/<your-knowledge-box-id>/search?query=Batman&features=document&features=paragraph

If your Knowledge Box is not public, you must provide the X-NUCLIA-SERVICEACCOUNT header with an API token or an Authorization header.

To search in a specific resource, the search endpoint path is:

https://<zone>.nuclia.cloud/api/v1/kb/<your-knowledge-box-id>/resource/<resource-id>/search

Reference documentation

The Nuclia API documentation is available here.

How to query a Knowledge Box

Search parameters​

Query​

Simple query​

Filters​

Advanced filtering​

all​

any​

none​

not_all​

Combining​

Date filtering​

Search in a specific field​

Minimum score​

BM25 scores​

Semantic scores​

Result options​

Features​

Facets​

Highlight matching words​

How to call the search endpoint​

Reference documentation​

Search parameters

Query

Simple query

Filters

Advanced filtering

all

any

none

not_all

Combining

Date filtering

Search in a specific field

Minimum score

BM25 scores

Semantic scores

Result options

Features

Facets

Highlight matching words

How to call the search endpoint

Reference documentation