TeksInfra: install the InfraOps worker, connect Airflow, dbt and Snowflake, approve fixes

TeksInfra is the autonomous operations and deployment layer of Tekspaz. While traditional DevOps relies on manual runbooks and 3 AM on-call paging, TeksInfra uses specialized InfraOps agents to monitor pipelines, ingest telemetry from Airflow, dbt, and Snowflake, identify root causes, and propose remediating pull requests.

Everything is dry-run by default until an engineer explicitly approves the fix.

---

1. Architecture of TeksInfra & the InfraOps worker

TeksInfra consists of two components:

1. The TeksInfra Platform: Central web dashboard in Tekspaz (/projects/[id]/infra) for visualizing pipelines, database telemetry, incident logs, and approval buttons.
2. The InfraOps Worker: A lightweight, secure background agent running in your cloud infrastructure or VM. It ingests local telemetry, talks to Airflow/dbt/Snowflake, evaluates alert playbooks, and reports findings to the platform via an outbox queue.

flowchart TD
    subgraph CustomerCloud["Customer Tenancy / VM"]
        AF["Apache Airflow"] -->|Failure Events| Worker["InfraOps Worker Daemon"]
        DBT["dbt Artifacts"] -->|Run Results| Worker
        SF["Snowflake / DB"] -->|Telemetry & Query History| Worker
        Worker -->|Alerts & Root Cause Analysis| Outbox["Outbox Queue"]
    end
    Outbox -->|Encrypted HTTPS| Platform["Teks1Platform / TeksInfra UI"]
    Platform --> Human{"Engineer Reviews PR / Fix"}
    Human -- "Approve Fix" --> Worker
    Worker -->|Execute Revert / SQL Patch| SF

---

2. Installing the InfraOps worker

The InfraOps worker can be installed on any Linux VM (Ubuntu 22.04+ or Debian 12+) or deployed as a container in Kubernetes.

Step 1: Install prerequisites

Ensure Python 3.11+ and systemd are installed:

sudo apt-get update && sudo apt-get install -y python3 python3-venv git curl

Step 2: Clone worker and install dependencies

git clone https://github.com/skilled-coderAI/teksinfra-worker.git /opt/infraops-worker
cd /opt/infraops-worker
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Step 3: Configure config/tenants.yml

Create or edit /opt/infraops-worker/config/tenants.yml:

version: "1.0"
tenant:
  id: "org_acme_corp"
  platform_url: "https://teks1platform.tekspaz.ai"
  api_key: "env:INFRAOPS_PLATFORM_KEY"
  dry_run: true # Leave true initially to verify recommendations safely

connectors:
  airflow:
    enabled: true
    webhook_port: 8085
  dbt:
    enabled: true
    artifacts_dir: "/var/log/dbt/target"
  snowflake:
    enabled: true
    account: "xy12345.us-east-1"
    warehouse: "FINOPS_WH"
    role: "INFRAOPS_ROLE"
    username: "TEKSINFRA_SVC"

Step 4: Configure systemd service

Create /etc/systemd/system/infraops-worker.service:

[Unit]
Description=Tekspaz InfraOps Worker
After=network.target

[Service]
Type=simple
User=root
WorkingDirectory=/opt/infraops-worker
EnvironmentFile=/etc/default/infraops-worker
ExecStart=/opt/infraops-worker/.venv/bin/python main.py run --config config/tenants.yml
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable infraops-worker
sudo systemctl start infraops-worker
sudo systemctl status infraops-worker

---

3. Connecting Apache Airflow

When a DAG run or task fails, Airflow sends the task execution context, logs, and stack trace to the InfraOps worker.

Setup Airflow Failure Callback

Add the Tekspaz failure callback to your Airflow environment or base DAG configuration:

from airflow.exceptions import AirflowException
import requests

def tekspaz_task_failure_alert(context):
    payload = {
        "dag_id": context.get("dag").dag_id,
        "task_id": context.get("task_instance").task_id,
        "execution_date": str(context.get("execution_date")),
        "try_number": context.get("task_instance").try_number,
        "log_url": context.get("task_instance").log_url,
        "exception": str(context.get("exception")),
    }
    try:
        requests.post(
            "http://127.0.0.1:8085/v1/airflow/events",
            json=payload,
            timeout=5
        )
    except Exception as e:
        print(f"Failed to post to InfraOps worker: {e}")

# In your default_args:
default_args = {
    'owner': 'data-team',
    'on_failure_callback': tekspaz_task_failure_alert,
}

The InfraOps worker ingests the event, extracts the SQL or Python error, identifies upstream schema changes, and drafts an incident in TeksInfra.

---

4. Connecting dbt artifacts

TeksInfra parses dbt build results to detect failing models, test assertion errors, and schema contract breaches.

Configuration

In your CI/CD or Airflow pipeline after running dbt build:

# Push dbt artifacts to the worker intake
curl -X POST http://127.0.0.1:8085/v1/dbt/artifacts \
  -H "Content-Type: application/json" \
  -F "manifest=@target/manifest.json" \
  -F "run_results=@target/run_results.json"

The worker analyzes the directed acyclic graph (DAG) of your models, isolates the root-cause failure (e.g. fct_daily_orders), and ignores false-positive downstream errors.

---

5. Connecting Snowflake telemetry

To monitor warehouse health and query anomalies:

1. Create a dedicated read-only service user and role in Snowflake:

   CREATE ROLE IF NOT EXISTS INFRAOPS_ROLE;
   GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE TO ROLE INFRAOPS_ROLE;
   GRANT MONITOR ON WAREHOUSE COMPUTE_WH TO ROLE INFRAOPS_ROLE;
   
   CREATE USER IF NOT EXISTS TEKSINFRA_SVC
     PASSWORD = '...'
     DEFAULT_ROLE = INFRAOPS_ROLE;
   GRANT ROLE INFRAOPS_ROLE TO USER TEKSINFRA_SVC;

2. Set the credentials in /etc/default/infraops-worker:

   SNOWFLAKE_USER="TEKSINFRA_SVC"
   SNOWFLAKE_PASSWORD="<STRONG_PASSWORD>"
   SNOWFLAKE_ACCOUNT="<ORG-ACCOUNT>"

3. Restart the worker: sudo systemctl restart infraops-worker.

---

6. Incident triage and approving fixes

When an incident is detected (e.g., INC-RUNAWAY, INC-IDLE, ALWAYS_ON, or INC-CASCADE):

1. Alert Notification: The on-call channel (Slack or Teams) receives an alert card detailing the warehouse, severity, root-cause hypothesis, and proposed action.
2. Review in TeksInfra: Click the alert link to open TeksInfra › Activity (/projects/[id]/infra).
3. Inspect the Remediation:
   • View the generated SQL patch or GitHub Pull Request.
   • For runaway queries: inspect KILL QUERY <id> statement and affected warehouse metrics.
   • For dbt model failure: inspect proposed SQL casting fix and diff.
4. Approve Fix: Click Approve & Apply.
   • If dry_run: true, the command prints the dry-run simulation output.
   • If dry_run: false, the worker executes the remediation and verifies that health signals recover within 5 minutes.

---

What to explore next

• TeksOptima: connect Snowflake and Databricks, read the savings
• API and CLI for developers
• FAQ and troubleshooting