Skip to main content

QSC Feeding Pipeline Reader

FlatJsonFeedingQueueReader​

The FlatJsonFeedingQueueReader is a reader component that processes flat JSON data and converts it into documents suitable for indexing in QSC.

name: "full push api reader for flat json"
resumeable: true
reader:
  name: "FlatJsonFeedingQueueReader"
  clazz: "com.quasiris.qsc.feeding.FlatJsonFeedingQueueReader"

Example to index the flat json to elastic​

name: "full push api reader for flat json"
resumeable: true
reader:
  name: "FlatJsonFeedingQueueReader"
  clazz: "com.quasiris.qsc.feeding.FlatJsonFeedingQueueReader"
filter:
  - name: "Status filter"
    clazz: "solrcmdutils.StatusTimeFilter"
    disabled: false
    property:
      webHook: "{{{ jobBaseUrl}}}/api/v1/job/{{{ processId }}}/_status-update?status=${status}&documentCounter=${documentCounter}&expectedDocumentCount={{{ expectedDocumentCount }}}"
 {{#elasticIndexUrls}}
   - name: "Elastic Writer"
     clazz: "solrcmdutils.ElasticWriter"
     disabled: false
     property:
       bulkMethodFieldName: "operation"
       includeTypeName: "false"
       detectNumberValues: "false"
       idField: "docId"
       type: "elastic"
       delete: "false"
       housekeepingEnabled: "{{{ housekeepingEnabled }}}"
       elasticMappingLocation: "{{{ elasticSettingsUrl }}}"
       location: "{{{ . }}}/_doc"
       bufferSize: "50"
 {{/elasticIndexUrls}}