Skip to content

Add Xquik read-only X data tools#6385

Open
kriptoburak wants to merge 2 commits into
FlowiseAI:mainfrom
kriptoburak:feature/xquik-read-tools
Open

Add Xquik read-only X data tools#6385
kriptoburak wants to merge 2 commits into
FlowiseAI:mainfrom
kriptoburak:feature/xquik-read-tools

Conversation

@kriptoburak
Copy link
Copy Markdown

@kriptoburak kriptoburak commented May 14, 2026

Description

Adds a Flowise Tools node for Xquik read-only public X/Twitter data, scoped to the discussion feedback in #6374 (comment).

  • Adds an Xquik API credential using a password field.
  • Exposes selected read-only tools for Search Tweets, Get Tweet, Get User, Search Users, List User Tweets, and Get Trends.
  • Adds provenance metadata to every returned item: source, resource_type, id, url, author_id, created_at, retrieved_at, query, and rate_limit.
  • Preserves pagination metadata with has_next_page and next_cursor when Xquik returns it.
  • Keeps posting, likes, follows, DMs, and profile updates out of this first version.
  • Reads response bodies as text before JSON parsing and fails fast on invalid JSON or unsupported payload shapes.

Validation

  • Focused Xquik Jest test passed: 10 tests.
  • flowise-components build passed through TypeScript and gulp.
  • Targeted eslint passed for the new credential and Xquik tool files.
  • Whitespace check passed.
  • Link check passed for the discussion URL, Xquik overview docs, and x-twitter-scraper repo URL.

Note: full flowise-components lint still reports existing unused eslint-disable directives outside this change in ChatFireworks, GoogleGenerativeAIEmbedding, CustomTool, OpenAPIToolkit, and validator. The new Xquik files pass targeted lint.

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the Xquik X Data toolset, enabling the retrieval of read-only public X/Twitter data such as tweets, users, and trends, complete with provenance metadata for RAG workflows. The implementation includes a new credential class, core tool logic, and unit tests. Feedback from the review focuses on improving the robustness of API response handling; specifically, it is recommended to read response bodies as text before parsing to prevent stream consumption errors in node-fetch and to adopt a fail-fast strategy by throwing errors when encountering invalid payload formats.

Comment on lines +321 to +331
async function readResponsePayload(response: Response): Promise<unknown> {
try {
return await response.json()
} catch {
try {
return await response.text()
} catch {
return undefined
}
}
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

In node-fetch, the response body can only be consumed once. If response.json() fails because the response is not valid JSON, the body stream is already consumed. It is safer to read the body as text first and then attempt to parse it as JSON. Additionally, per our fail-fast policy for external data, we should throw an error if the response cannot be parsed rather than returning a default or empty value.

async function readResponsePayload(response: Response): Promise<unknown> {
    const text = await response.text()
    try {
        return JSON.parse(text)
    } catch (err) {
        throw new Error("Failed to parse response as JSON from external source")
    }
}
References
  1. When handling potentially invalid data from external sources (like an API response), prefer throwing an error for invalid input types rather than silently returning a default or empty value. This promotes fail-fast behavior.

Comment thread packages/components/nodes/tools/Xquik/core.ts
@kriptoburak
Copy link
Copy Markdown
Author

Updated in bab3c0d.

  • Response bodies are now read as text first, then parsed as JSON, so node-fetch streams are consumed once.
  • Invalid JSON and empty response bodies now fail fast.
  • Unsupported payload shapes now fail fast instead of returning an empty result.
  • Top-level array payloads are supported and covered by tests.
  • Focused Xquik Jest now passes 10 tests, targeted eslint passes, and flowise-components build passes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant