Email and HTTP Ingress

Two additional file sources cover pull-based ingress patterns that don't fit the SFTP/cloud-storage model: polling a remote HTTP API for available files (HttpPullSource) and ingesting email attachments from an IMAP mailbox (EmailInboxSource).

Neither supports event-driven mode — both use polling only.

---

HTTP Pull source

HttpPullSource polls a remote API endpoint for a list of downloadable files and fetches each one. It handles bearer token, API key header, and OAuth2 client credentials authentication.

Expected API contract

The list_url endpoint must return JSON with a files array:

{
  "files": [
    {
      "id": "abc123",
      "url": "https://partner.example.com/exports/payments-2024-01-15.csv",
      "name": "payments-2024-01-15.csv",
      "size": 48291,
      "modified_at": "2024-01-15T09:00:00Z",
      "etag": "d41d8cd98f00b204e9800998ecf8427e"
    }
  ]
}

Required fields per file entry: url, name. Optional: id, size, modified_at, etag, content_type, metadata.

Authentication

Bearer token

from craft_easy_file_import.sources.http_pull import HttpPullSource

source = HttpPullSource(
    name="partner-api",
    list_url="https://partner.example.com/api/exports",
    credential_name="partner_api_token",  # raw token string in credential store
    auth_type="bearer",
)

Sends Authorization: Bearer <token> on every request.

API key header

source = HttpPullSource(
    name="partner-api",
    list_url="https://partner.example.com/api/exports",
    credential_name="partner_api_key",
    auth_type="api_key_header",
    api_key_header_name="X-API-Key",  # default
)

OAuth2 client credentials

source = HttpPullSource(
    name="partner-api",
    list_url="https://partner.example.com/api/exports",
    credential_name="partner_oauth_creds",
    auth_type="oauth2_client_credentials",
    token_url="https://partner.example.com/oauth/token",
)

The credential stored under credential_name must be JSON with client_id and client_secret (and optionally scope):

{"client_id": "my-client", "client_secret": "s3cr3t", "scope": "imports:read"}

Tokens are cached in memory and refreshed 60 seconds before expiry (expires_in from the token response).

Download timeout

Large files may require a longer timeout than the list call:

source = HttpPullSource(
    name="partner-api",
    list_url="...",
    credential_name="...",
    download_timeout=300.0,  # seconds, default 300
)

Move and delete

HTTP pull sources are read-only — move_file() and delete_file() are no-ops that log a warning. The source has no way to manage files on the remote server. The runner's post-processing (completed/error folder moves) has no effect for this source type.

---

Email inbox source

EmailInboxSource polls an IMAP mailbox for unseen messages, verifies sender authentication (DKIM and SPF), and extracts file attachments for import processing.

pip install 'craft-easy-file-import[email]'

Basic configuration

from craft_easy_file_import.sources.email_inbox import EmailInboxSource

source = EmailInboxSource(
    name="invoice-inbox",
    imap_host="imap.example.com",
    imap_port=993,                # default, TLS
    credential_name="imap_creds",
    mailbox="INBOX",
    quarantine_mailbox="quarantine",
    allowed_senders=["partner@supplier.com", "*.erp-provider.com"],
    require_dkim=True,
    require_spf=True,
    attachment_regex=r".*\.(csv|json|xml|xlsx)$",
)

Credentials stored under credential_name must be JSON with username and password:

{"username": "imports@example.com", "password": "app-password"}

Sender allowlist

allowed_senders supports exact addresses and wildcard domains:

Pattern	Matches
partner@supplier.com	That exact address only
*.erp-provider.com	Any address ending in @erp-provider.com
(empty list)	All senders allowed

Emails from addresses not in the allowlist are silently skipped (logged as email_rejected_unauthorized_sender).

DKIM and SPF verification

DKIM and SPF are checked via the Authentication-Results header written by the receiving MTA — the source does not perform cryptographic DKIM verification itself. This is intentional: MTA-level verification runs before the message reaches the mailbox, so by the time the source polls it, the result is already in the header.

• require_dkim=True — rejects messages unless Authentication-Results contains dkim=pass.
• require_spf=True — rejects messages unless Authentication-Results contains spf=pass.

Messages that fail DKIM or SPF verification are moved to quarantine_mailbox (not deleted) and marked as seen. Log events: email_quarantined_dkim_failed, email_quarantined_spf_failed.

If your MTA does not write Authentication-Results, set both to False:

EmailInboxSource(
    ...,
    require_dkim=False,
    require_spf=False,
)

Attachment filtering

Only attachments whose filename matches attachment_regex are extracted. The default pattern accepts common data exchange formats:

.*\.(csv|json|xml|dat|txt|xlsx|xls|tsv|zip|gz)$

Attachments without a filename get a synthetic name: attachment_{idx}.{mime_subtype}.

Staging

Extracted attachments are held in an in-memory staging dict ({staged_path: bytes}). The runner calls source.open(staged_path) to read them. Staged paths use the format:

email_inbox/{message_id_sanitized}/{idx}_{filename}

Each FileInfo returned by poll() includes metadata from the original email in custom_metadata:

{
    "email_message_id": "<abc@example.com>",
    "email_from": "partner@supplier.com",
    "email_subject": "Weekly export 2024-W03",
    "email_received_at": "Mon, 15 Jan 2024 09:00:00 +0000",
}

move_file() renames within the staging dict. delete_file() removes from the staging dict. Neither touches the IMAP server — emails are marked as seen during poll() so they are not re-processed on the next cycle.

Disabling DKIM/SPF in development

Useful when testing with a local SMTP/IMAP server that does not set Authentication-Results:

source = EmailInboxSource(
    name="local-test",
    imap_host="localhost",
    imap_port=993,
    credential_name="test_imap",
    require_dkim=False,
    require_spf=False,
    allowed_senders=[],  # accept all
)