Skip to main content

Getting Started Feeding

To make data searchable the data must be preprocessed and indexed in the search index. To import the data into the QSC several feeding mechanisms are available.

  • Bulk Feeding API
  • Sitemap Crawling

In the following examples we will use

  • tenant-code: demo
  • feeding-code: products

Preconditions

  • A feeding configuration is already configured in the QSC (TODO link to documentation for configuring a feeding)

Fullfeed​

Sometimes it happens that update or delete events get lost. An update can be send again and the index is in sync. For a lost delete event, often the document is not available anymore in the source system. In this case it is not possible to trigger the delete again.

With a fullfeed it is possible to sync all data and remove not available documents.

  • start the fullfeed POST /api/v1/data/fullfeed/start/{tenantCode}/{feedingCode}
  • feed all documents via the bulk api POST /api/v1/data/bulk/qsc/{tenantCode}/{feedingCode}
  • end the fullfeed - POST /api/v1/data/fullfeed/end/{tenantCode}/{feedingCode}
  • cancel the fullfeed if the full feed fails - POST /api/v1/data/fullfeed/cancel/{tenantCode}/{feedingCode}

Bulk Feeding API​

With the bulk feeding API it is possible to send multiple documents in a batch.

  • A good batch size to begin with is 100
    • For small documents, the batch size can be increased to improve performance
  • It is possible to mix update and delete events
  • Documents are processed in the same order as they were sent to the API
    • that means, if you send for example an update and a delete, the document will be deleted

Feeding types​

  • qsc: to feed documents for search
  • reporting: to feed reporting data

Feed documents via Bulk Feeding API​

In order to feed a batch of documents to the bulk feeding API you have to perform an HTTP POST

  • URL
    • Structure:https://qsc.quasiris.de/api/v1/data/bulk/[feeding-type]/[tenant-code]/[feeding-code]
    • In our example: https://qsc.quasiris.de/api/v1/data/bulk/qsc/demo/products
  • Authentication (should be configured in securityIds of a feeding configuration)
    • api key authentication. X-QSC-Token: your API token e.g. bbc20990-704f-45bb-9f56-e80bca8d0c86
      • The API is secured by an API Token which must be sent as a header value!
    • basic authentication
    • public authentication
  • Headers:
    • Content-type: application/json
  • Payload:
    • The payload is a list of documents in JSON with the data structure described below
curl -X POST https://qsc-dev.quasiris.de/api/v1/data/bulk/qsc/demo/products -H "Content-Type: application/json" -H "X-QSC-Token: bbc20990-704f-45bb-9f56-e80bca8d0c89" -d '
[
	{
		"header": {
            "id": "0815",
            "action": "update"
		},
		"payload": {
			"id": "0815",
			"title": "bar"
		}
	}
]'

Data structure​

A document in the bulk API consists of a header and a payload.

The header contains a unique id, and an action to update or delete a document:

  • "id": unique identifier for the document
  • "action": either "delete" or "update"
Header example: Update event​
"header": {
    "id" : "0815",
    "action" : "update"
}
Header example: Delete event​
"header": {
    "id" : "0815",
    "action" : "delete" 
}

Payload​

The data types for each field are configured in the Mapping tab of the feeding.value

data types:

  • string
  • list.string
  • number
  • long
  • integer | int
  • double
  • boolean
  • date (iso format - 2021-09-20T08:11:24.372Z)
  • attributes
  • categories
  • categoryPath
  • raw

The payload contains the data as a key value pairs:

  • key: name of a field
  • value: single value or multi value of the field
    • single value: string, long, double, boolean
    • multi value: an array of string, long, double or boolean
    • json: a json array or json value
Payload example​

single value examples:

  • "title" (string),
  • "price" (double, number)
  • "availability" (boolean)
  • "jsonExample" (raw)
  • "updatedAt" (date in iso format)

multi value examples

  • "keywords" (list.string)
"payload: {
    "title" : "iphone 12",
    "price" : 1295.99,
    "availability" : true,
    "updatedAt" : "2021-09-20T08:11:24.372Z",
    "keywords" : [
        "smartphone",
        "telephone"
    ],
    "jsonExample" : {
        "foo" : "bar"
     }

}

Categories (type: categories)​

In the "categories" section multiple categories of a document can be specified.

  • this section can be used by specifying "categories" as key in the document payload
    • the value of this section is an array of categories in the specific structure described below
  • a document can exists in multiple categories

Properties​

  • the "categories" array contains elements with key "category" and an array as value
    • the array contains elements, representing the path from the root category, through the child categories up to the current leaf category
    • each of these elements contain the following properties:
      • "id": unique identifier for the category
      • "name": name of the category
      • "level": depth of the category, described below
Levels​

The category is referenced by traversing from the root category to the leaf category

  • root category has "level": 0
  • branch categories have levels between 0 and the level of the leaf category
  • leaf category is the child category with the highest level

Examples​

Example: Categories array with one category​

In the following example the product exists in one category:

  1. "bathroom" -> "showers" -> "shower stalls" -> "corner showers"
  • The root category is "bathroom"
  • The branch categories are "showers" and "shower stalls"
  • The leaf category is "corner showers"
"categories": [
    {
        "category": [
            {
                "id": "d5245409d00c4a969dd28eb3d183267b",
                "name": "bathroom",
                "level": 0
            },
            {
                "id": "d6298a2bba6c478f8b1422fc81b0658c",
                "name": "showers",
                "level": 1
            },
            {
                "id": "bb1eaed512284530a5b5c077a07a25eb",
                "name": "shower stalls",
                "level": 2
            },
            {
                "id": "ccbf43b6ab634ffba7dcd62b50a6ee1a",
                "name": "corner showers",
                "level": 3
            }
        ]
    }
]
Example: Categories array two categories​

In the following example the product exists in two categories

  1. "bathroom" -> "showers" -> "shower stalls" -> "corner showers"
  2. "swimming pool" -> "showers"
"categories": [
    {
        "category": [
            {
                "id": "d5245409d00c4a969dd28eb3d183267b",
                "name": "bathroom",
                "level": 0
            },
            {
                "id": "d6298a2bba6c478f8b1422fc81b0658c",
                "name": "showers",
                "level": 1
            },
            {
                "id": "bb1eaed512284530a5b5c077a07a25eb",
                "name": "shower stalls",
                "level": 2
            },
            {
                "id": "ccbf43b6ab634ffba7dcd62b50a6ee1a",
                "name": "corner showers",
                "level": 3
            }
        ]
    },
    {
        "category": [
            {
                "id": "c0cd8883413c48318db1658e99f88a20",
                "name": "swimming pool",
                "level": 0
            },
            {
                "id": "73c5633392734b59b7c6f93e1b595368",
                "name": "showers",
                "level": 1
            }
        ]
    }
]

Validation rules for categories​

  • level for the same category must be unique
  • Single category only has (id, name, level) fields
  • id is a String field. It is not null, and it is not blank
  • name is a String field. It is not null, and it is not blank
  • level is a Long numeric field. It can be 0 or greater

Categories as Path (type: categoryPath)​

The categories can be also übergeben as a path.categoryPath

{
    "myCategory": "Bekleidung/Damen/Oberteile"
}

Attributes (type: attributes)​

In the "attributes" section multiple dynamic attributes for a product are transmitted.

  • this section can be used by specifying "attributes" as key in the document payload
    • the value of this section is an array of attributes in the specific structure described below
  • a document can have multiple attributes

Properties​

Each attribute element in "attributes" can contain the following properties:

  • "id": a unique identifier of the attribute
    • if "id" is missing, it is automatically generated from "name"
  • "name": the name of the attribute that is used for displaying in the frontend
  • "values": the list of values
  • "min": the minimum value of a range
  • "max": the maximum value of a range
  • "unit": unit (eg. mm, cm, g, kg, ...)
  • "dataType": string, double, long, boolean, date
  • "position": The position of the attribute in the detail page

Examples​

The following examples show some typical attribute use cases:

Example: "width" as double value range​
{
    "id": "width",
    "name": "Width",
    "min": 1,
    "max": 99,
    "unit": "cm",
    "dataType": "double",
    "position": 1
}
Example: "width" as single double value​
{
    "id": "width",
    "name": "Width",
    "values" : [
      55
    ],
    "unit": "cm",
    "dataType": "double",
    "position": 1
}
Example: "color" as string value​
{
    "id": "color",
    "name": "color",
    "values": [
        "red"    
    ],
    "dataType": "string",
    "position": 1
}
Example: "portable" as boolean value​
{
    "id": "portable",
    "name": "portable",
    "values": [
        false    
    ],
    "dataType": "boolean",
    "position": 1
}
Example: "validPeriod" as date value range​
{
    "id": "validPeriod",
    "name": "valid Period",
    "min": "2021-09-20T08:11:24.372Z",
    "max": "2021-09-20T08:11:24.372Z",
    "dataType": "date",
    "position": 1
}
Example: Attributes array with two attributes​
"attributes": [
	{
		"id": "contents",
		"name": "Package Contents",
		"values": [
			"Manual",
			"Power Cable",
			"2x AA Batteries"
		],
		"dataType": "string"
	},
	{
		"id": "01-dia",
		"name": "Diameter",
		"values": [
			127
		],
		"dataType": "long",
		"unit": "mm"
	}
]

Validation Rules for Attributes​

  1. "id" can be null or a not blank unique string
  2. "name" can be null or a not blank string
  3. "dataType" can be "string", "boolean", "long", "double" or "date"
    1. For "string" and "boolean" data types the "values" array is required
    2. For "long", "double" and "date" data types you can choose between simple values and range fields
      1. You must not combine range values and an "values" array in one attribute
      2. To use simple values, use "values" array
      3. To use range fields, use "min", "max" properties
        • "min" field value must be less or equals to "max" field value
        • e.g. for "long" data type 1 < 2
        • e.g. for "date" data type "2020-08-06" < "2020-08-07"
  4. If you specify the "values" array, it must not be empty an array
  5. "position" field is optional
    1. can be null, or an integer value
Acceptable values for data types​
Specified "dataType"acceptable values in "values"
"string"any value in string format
"boolean"true, false and also "true", "false" in string format
"long"integer raw values (1, 2) or integer values in string format ("1", "2")
"double"are float or integer raw values (1.4, 2) or the same type of values in string format ("1.4", "2")
"date"dates in string format
Supported formats for "date" dataType​
  • "2020-08-06"
  • "2007-12-03T10:15:30.00Z"
  • "2020-08-06T22:18:26+0000"
  • "2020-08-06T22:18:26.528+0000"
  • "2020-08-06T22:18:26.528+00:00"

Example: Bulk Feeding API Usage​

In the following example a document with "id" "0815" is updated, by sending a list containing just this document to the Bulk Feeding API. The updated document contains the fields "id", "title", "attributes" and "categories":

curl -X POST https://qsc-dev.quasiris.de/api/v1/data/bulk/qsc/demo/products -H "Content-Type: application/json" -H "X-QSC-Token: bbc20990-704f-45bb-9f56-e80bca8d0c89" -d '
[
	{
		"header": {
			"id": "0815",
			"action": "update"
		},
		"payload": {
			"id": "0815",
			"title": "bar",
			"attributes": [
				{
					"id": "farbe",
					"name": "Farbe",
					"values": [
						"weiß"
					],
					"dataType": "string"
				},
				{
					"id": "breite",
					"name": "Breite",
					"values": [
						120
					],
					"dataType": "long",
					"unit": "mm"
				},
				{
					"id": "hoehe",
					"name": "Höehe",
					"min": 50,
					"max": 100,
					"dataType": "double",
					"unit": "cm"
				}
			],
			"categories": [
				{
					"category": [
						{
							"id": "d5245409d00c4a969dd28eb3d183267b",
							"name": "example products",
							"level": 0
						},
						{
							"id": "d6298a2bba6c478f8b1422fc81b0658c",
							"name": "foo",
							"level": 1
						}
					]
				}
			]
		}
	}
]'