Skip to main content

QSC Feeding Pipeline

Resume​

To resume a feeding pipeline set the property resumeable: true in pipeline configuration.

name: "full push api reader"
resumeable: true
reader:
  name: "QscFeedingQueueReader"
  clazz: "com.quasiris.qsc.feeding.QscFeedingQueueReader"
filter:
  - name: "Status filter"
    clazz: "solrcmdutils.StatusTimeFilter"
    disabled: false
    property:
      webHook: "{{{ jobBaseUrl}}}/api/v1/job/{{{ processId }}}/_status-update?status=${status}&documentCounter=${documentCounter}&expectedDocumentCount={{{ expectedDocumentCount }}}"

QscFeedingQueueReader​

  • com.quasiris.qsc.feeding.QscFeedingQueueReader

Ignore Errors​

When we use push pipeline type with default datasource we can specify it using ignoreErrorsCount variable in feeding configuration. Or if we use custom pipeline type we can customize it by ignoreErrorsCount property of QscFeedingQueueReader.

Example (for custom pipeline):

reader:
  name: "QscFeedingQueueReader"
  clazz: "com.quasiris.qsc.feeding.QscFeedingQueueReader"
  property:
    ...
    ignoreErrorsCount: "-1"
    ...

Example for variables:

"variables": {
    ...
    "ignoreErrorsCount": -1,
    ...
},
  • 0 (default) strict - fail on first error
  • negative value - ignore all errors
  • positive values specify number of errors to ignore

Mask passwords​

When configuring and running pipelines, sensitive information—such as passwords—may inadvertently appear in build logs. To prevent exposing these credentials, you can mask them by applying the mask(...) function. Below is a step-by-step guide on how to configure passwords in a pipeline and ensure they are masked in logs.

Using Security​

  • $security.securityConfigName.secret can be resolved in feeding configs
  • $security.securityConfigName.securityType can be resolved in feeding configs
  • Control access by grants (the same like in config)