Skip to main content
The file connector lets you send jobs without hosting files in S3 or publishing temporary URLs. Upload files once, reference their IDs in a connector definition, and reuse them across multiple jobs until they expire.

​
When to use the file connector

  • You need to process documents or videos that only exist locally.
  • You want to avoid managing temporary URLs or granting the service access to your buckets.
  • You plan to upload once and run multiple jobs against the same inputs.
Uploaded files expire after the retention period you set (30 days by default). Jobs fail if a referenced file has already expired.

​
Workflow overview

  1. Upload files and capture the returned IDs.
  2. (Optional) List files to review which IDs remain available.
  3. Create jobs with a connector payload that lists one or more file_ids.

​
Upload files

Upload a file with optional expiration. The API stores the object under your account and returns a unique ID.
  • TypeScript
  • Python
import { DataIngestionClient } from "@trelent/data-ingestion";
import { readFileSync } from "fs";

const client = new DataIngestionClient();

const fileBuffer = readFileSync("sample.pdf");
const blob = new Blob([fileBuffer], { type: "application/pdf" });

const upload = await client.uploadFile(blob, "sample.pdf", { expiresInDays: 7 });
console.log("File ID:", upload.id);

{
  "id": "e2d4c3d2-41c0-4a6b-a387-580807fb5ad2"
}
​
file
bytes / Blob
required
Binary payload of the document or video you want to ingest.
​
filename
string
required
Name of the file including extension.
​
expires_in_days / expiresInDays
integer
default:"30"
Number of days the upload should remain available. Lower values reduce your storage footprint.

​
Create jobs with uploaded files

The connector definition references the files you uploaded. Order is preserved, so the first file ID in the list becomes the first input identifier in job responses.
​
connector.type
string
required
Set to file_upload.
​
connector.file_ids
array<string>
required
List of uploaded file IDs you want to process. You must own every referenced ID.
  • TypeScript
  • Python
import { DataIngestionClient } from "@trelent/data-ingestion";
import type { JobInput } from "@trelent/data-ingestion";

const client = new DataIngestionClient();

const job: JobInput = {
  connector: {
    type: "file_upload",
    file_ids: [
      "e2d4c3d2-41c0-4a6b-a387-580807fb5ad2",
      "b4c77b85-3ad4-4cb0-8d06-5ff8ff5df2ad",
    ],
  },
  output: {
    type: "s3-signed-url",
    expires_minutes: 120,
  },
};

const response = await client.submitJob(job);
console.log("Job ID:", response.job_id);

​
List available uploads

List your uploads to see which IDs remain valid or to map friendly filenames to IDs. Each entry includes the filename and creation timestamp.
  • TypeScript
  • Python
import { DataIngestionClient } from "@trelent/data-ingestion";

const client = new DataIngestionClient();

const files = await client.listFiles();
for (const file of files.files) {
  console.log(file.id, file.filename, file.created_at);
}

{
  "files": [
    {
      "id": "e2d4c3d2-41c0-4a6b-a387-580807fb5ad2",
      "filename": "sample.pdf",
      "created_at": "2024-11-18T22:15:05.197Z"
    }
  ]
}
Need a full walkthrough? Follow the quickstart to upload a file and process it end-to-end.