How to export QSC Feeding Data

This documentation covers the usage of the Python script designed to export data from a specific API endpoint and write each entry as a separate line in a JSON Lines (JSONL) file.

Overview

The script fetches data from https://qsc-dev.quasiris.de/api/v1/feeding/wins/wiki/_data-full, handling pagination and custom headers. Each entry from the returned data array is written as a new line in a .jsonl file.

Requirements

• Python 3
• requests library

Installation

Before running the script, ensure that you have Python 3 installed on your machine. You also need to install the requests library, which can be done using pip:

pip install requests

Usage

To use the script, you need to provide the API token and specify the output file path. Replace YOUR_X_QSC_TOKEN_HERE with your actual X-QSC-Token.

Script Parameters

• url: The API endpoint URL.
• token: The token required for authentication with the API, passed as a custom header (X-QSC-Token).
• output_path: Path to the output .jsonl file where the data will be saved.

Running the Script

Execute the script with Python. The script fetches data from the given URL, paginating through the results, and writes each entry to the specified output file in JSON Lines format.

Output Format

The output file will be in JSON Lines format, where each line is a valid JSON object representing an individual entry from the API response.

Example

# Set your parameters
url = 'https://qsc-dev.quasiris.de/api/v1/feeding/wins/wiki/_data-full'
token = 'YOUR_X_QSC_TOKEN_HERE'
output_path = 'exported_data.jsonl'

# Run the script
export_data_to_jsonl(url, token, output_path)

Troubleshooting

• Ensure the API token is correct.
• Check for network connectivity issues.
• Validate if the API endpoint is correct and accessible.

Code

import requests
import json

def fetch_data(url, page=1, token=''):
    """Fetch data from the given URL with pagination and a custom token."""
    headers = {'X-QSC-Token': token}
    params = {'page': page, 'size' : 100}
    response = requests.get(url, headers=headers, params=params)
    if response.status_code == 200:
        return response.json()
    else:
        print(f"Failed to fetch data for page {page}: {response.status_code}")
        return None

def export_data_to_jsonl(base_url, token, output_file):
    """Export data from the URL to a JSONL file."""
    page = 0

    with open(output_file, 'w') as file:
        while True:
            data = fetch_data(base_url, page=page, token=token)
            if len(data) == 0:
                break
            for entry in data:
                payload = json.loads(entry["payload"])
                json.dump(payload, file)
                file.write('\n')
            page += 1

    print(f"Data exported to {output_file}")


url = 'https://qsc-dev.quasiris.de/api/v1/feeding/wins/wiki/_data-full'
token = '**********'  # Replace witeUKgEZCvkjCEKjAvCAzfinTuxIPMpQh your actual X-QSC-Token
output_path = 'exported_data.json'
export_data_to_jsonl(url, token, output_path)