JCAM Integration — Data Flow Diagrams

JCAM is the DoJ-developed Cyber Security Assessment and Management platform (the
successor naming of CSAM). Each agency runs its own JCAM instance, so nothing in
this integration is tied to a particular host: the CLI reads jcamURL from
init.yaml, appends jcamBasePath (default /CSAM/api — unchanged by the CSAM →
JCAM rename), and versions each call per-path (v1/systems).

Every diagram below is drawn from regscale/integrations/public/jcam/. Field-level
mappings and setup steps are in jcam-integration.md; this
file is the shape of the data movement.

1. High level: JCAM ⇄ RegScale

flowchart LR
    subgraph JS["JCAM — agency-hosted"]
        JCAM[("JCAM API<br/>jcamURL + jcamBasePath<br/>e.g. https://jcam.yourdomain.gov/CSAM/api/v1/*")]
    end

    subgraph CLI["RegScale CLI — regscale/integrations/public/jcam"]
        SDK["sdk/ — JcamClient<br/>auth · retry · pagination · typed errors"]
        COMMON["common.py — shared client, jcamFilter,<br/>drift detection, run report"]
        MODULES["domain modules — front_matter, authorization,<br/>privacy, continuity, pocs, poam, controls,<br/>assessments, artifacts, interconnections, parameters"]
    end

    subgraph RS["RegScale"]
        RSAPI["RegScale REST / GraphQL"]
        ENT["SecurityPlan · Issue · Assessment · File<br/>User · SystemRole · FormFieldValue"]
    end

    JCAM <-->|"HTTPS · bearer token · HTTP/2"| SDK
    SDK <--> COMMON
    COMMON <--> MODULES
    MODULES <--> RSAPI
    RSAPI <--> ENT

    classDef jcam fill:#fde2e2,stroke:#a33,color:#000
    classDef int fill:#e8f5e9,stroke:#2e7d32,color:#000
    classDef rs fill:#dfe8ff,stroke:#335,color:#000
    class JCAM jcam
    class SDK,COMMON,MODULES int
    class RSAPI,ENT rs

2. Configuration → client construction

Nothing reaches JCAM until resolve_from_app_config produces a ClientConfig. When
it cannot, the run does not stop — the legacy helpers degrade to empty results,
which is why a misconfigured token looks like "JCAM returned nothing".

flowchart TD
    Y["init.yaml / environment"] --> R["sdk/auth.py<br/>resolve_from_app_config()"]
    R --> T{"jcamToken set and<br/>not a placeholder?"}
    T -- no --> E1["JcamConfigError<br/>'No JCAM Token in init.yaml'"]
    T -- yes --> U{"jcamURL set and<br/>not a placeholder?"}
    U -- no --> E2["JcamConfigError<br/>'No JCAM URL in init.yaml'"]
    U -- yes --> B["base_path = jcamBasePath when set,<br/>else /CSAM/api"]
    B --> C["ClientConfig — frozen, slots<br/>timeout 60s · connect 10s<br/>pool 50 · keepalive 20 / 30s<br/>http2 True · verify = sslVerify"]
    C --> S{"shared client already built for<br/>this (base_url, token)?"}
    S -- yes --> REUSE["reuse the process-wide JcamClient<br/>— pool, HTTP/2 session and retry<br/>controller are shared"]
    S -- no --> NEW["build one under _client_lock<br/>closing any prior client"]

    E1 --> W["_get_client() logs a warning<br/>and returns None"]
    E2 --> W
    W --> DEG["retrieve_/post_/put_to_jcam return []<br/>— the command keeps running with no data"]

    classDef bad fill:#fde2e2,stroke:#a33,color:#000
    class E1,E2,W,DEG bad

close_shared_client() runs at CLI teardown (the _with_api_cleanup decorator on
every command) to release the pool.

3. Single request lifecycle

sequenceDiagram
    participant M as domain module
    participant H as retrieve_/post_/put_to_jcam
    participant C as JcamClient.request
    participant R as tenacity Retrying
    participant X as httpx — pooled, HTTP/2
    participant J as JCAM API

    M->>H: relative path, e.g. "v1/systems/{id}/pocs"
    H->>C: get / post / put
    C->>C: _build_url — urljoin(base_url, base_path, path)
    C->>R: _send()
    R->>X: request
    X->>J: HTTPS + Authorization header
    J-->>X: 2xx / 4xx / 429 / 5xx

    alt status in 408, 429, 500, 502, 503, 504
        R->>R: Retry-After when parseable (capped at 30s),<br/>else exponential backoff + up to 0.25s jitter
        R->>X: resend — 5 attempts total
    end

    C->>C: _raise_for_status → typed exception
    alt success
        C-->>H: parsed JSON, raw bytes, or None
        H-->>M: list / dict / bytes
    else 404 JcamNotFoundError
        H-->>M: [] — logged at debug, "no record" is routine
    else any other JcamError
        H-->>M: [] + logger.exception
    end
    Note over H,M: strict=True re-raises instead of returning []<br/>— callers use it to tell "absent" from "failed"

Error taxonomy

flowchart TD
    JE["JcamError"] --> JCE["JcamConfigError<br/>— missing / placeholder config"]
    JE --> JTE["JcamTransportError<br/>— connect, read, TLS failures"]
    JE --> JHE["JcamHTTPError<br/>— status, url, body snippet"]
    JHE --> JAE["JcamAuthError — 401 / 403"]
    JHE --> JNF["JcamNotFoundError — 404"]
    JHE --> JRL["JcamRateLimitError — 429, carries retry_after"]
    JHE --> JSE["JcamServerError — 5xx"]

Pagination

flowchart LR
    P["JcamClient.paginate(path)"] --> M{"mode"}
    M -- page --> PG["page=1,2,3… + limit / pageSize"]
    M -- offset --> OF["offset += page_size"]
    PG --> F["fetch a page — 200 items by default"]
    OF --> F
    F --> S{"len(page) < page_size?"}
    S -- no --> N["advance and fetch again"]
    N --> F
    S -- yes --> D["stop — last page"]

4. Command surface

flowchart LR
    G["regscale jcam<br/>(regscale csam is a deprecated alias)"]

    subgraph IMP["Import — JCAM → RegScale"]
        I1["import_ssp"]
        I2["import_controls"]
        I3["import_poam"]
        I4["import_assessments"]
        I5["import_artifacts"]
        I6["import_parameters<br/>--crosswalk-path --workbook-out<br/>--learn-from --dry-run"]
    end

    subgraph EXP["Export — RegScale → JCAM"]
        X1["export_ssp --yes --force"]
        X2["export_poams --yes --force"]
        X3["export_controls<br/>Future Feature stub — exits immediately"]
        X4["export_assessment<br/>Future Feature stub — exits immediately"]
    end

    subgraph UTIL["Utility — read-only"]
        U1["test_jcam<br/>(test_csam alias)"]
        U2["check_custom_fields"]
        U3["preview_poc_roles"]
    end

    G --> IMP
    G --> EXP
    G --> UTIL

    classDef stub fill:#f5f5f5,stroke:#999,stroke-dasharray:4 3,color:#000
    class X3,X4 stub

5. `jcamFilter` — the gate on every flow

Import and export both scope themselves through jcamFilter. Nothing validates the
keys, so a typo silently matches nothing and the run stops.

flowchart TD
    A["GET v1/systems"] --> B{"jcamFilter configured?"}
    B -- no --> ALL["every JCAM system is in scope"]
    B -- yes --> F["filter_list — exact, case-sensitive equality<br/>AND across keys · OR only inside a YAML list<br/>no type coercion"]
    F --> M{"any system matched?"}
    M -- yes --> SCOPE["scoped system list"]
    M -- no --> EXIT(["error_and_exit<br/>'No results match filter in JCAM'"])
    ALL --> SCOPE
    SCOPE --> RUN["the command proceeds<br/>against these systems only"]

    classDef bad fill:#fde2e2,stroke:#a33,color:#000
    class EXIT bad

An unrecognized key (Id instead of id, 800-53R5 instead of 800-53r5) reaches
the same error_and_exit as a genuinely empty result — the messages are
indistinguishable, so check the key spelling first.

6. Import flow — `import_ssp`

import_ssp is the orchestrator; the other import_* commands run one slice of it.
The RegScale ⇄ JCAM correlation runs entirely through the JCAM Id custom field
(falling back to CSAM Id on pre-rename tenants) and SecurityPlan.otherIdentifier.

flowchart TD
    S(["regscale jcam import_ssp"]) --> RF["resolve_jcam_custom_fields<br/>'JCAM Id' → 'CSAM Id' fallback"]
    RF --> MAP["retrieve_ssps_custom_form_map<br/>RegScale SSP id ⇄ JCAM system id"]
    MAP --> FIL["fetch_filtered_jcam_systems<br/>(see the jcamFilter gate)"]
    FIL --> SPLIT["get_sync_records →<br/>new · existing · missing"]
    SPLIT --> NEW["create_ssps — new SecurityPlans<br/>added to the map so downstream<br/>domains cover them too"]
    NEW --> FM["save_ssp_front_matter"]
    FM --> D1["update_ssp_agency_details<br/>/agencydefineddataitems"]
    D1 --> D2["import_jcam_authorization<br/>/securityauthorization"]
    D2 --> D3["import_jcam_privacy_info<br/>/privacy + /sorn"]
    D3 --> D4["import_jcam_contingency<br/>/continuityresponse + /continuitytest"]
    D4 --> D5["import_jcam_pocs<br/>/systempointsofcontact"]
    D5 --> D6["import_jcam_additional_status<br/>/status + /additionalstatus"]
    D6 --> D7["import_jcam_info_types<br/>/infotypes"]
    D7 --> D8["import_jcam_interconnections<br/>/interconnections"]
    D8 --> D9["import_jcam_parent_child<br/>/FISMArollup"]
    D9 --> FIN["finalize_run — per-domain summary"]
    SPLIT -.->|"JCAM Ids on RegScale SSPs that<br/>the filter did not return"| MISS["logged as 'not in the JCAM<br/>filtered results' — left untouched"]

Each domain records its own outcome, so one failing domain never aborts the rest:

flowchart LR
    D["a domain, per system"] --> A{"outcome"}
    A -->|"data written"| S["record_success"]
    A -->|"404 / empty / no JCAM Id"| K["record_skipped"]
    A -->|"JcamError"| F["record_failure(domain, context, error)"]

7. Custom fields — where an import quietly loses data

Custom fields must exist before an import runs, and creating them is UI-only
(Admin → Modules → Custom Fields). check_custom_fields resolves names to ids but
does not create or enforce them.

flowchart TD
    A["a domain module needs one tab's fields"] --> B["FormFieldValue.check_custom_fields<br/>(module, tab, field names)"]
    B --> C{"field exists in RegScale?"}
    C -- yes --> D["field name → form_field_id"]
    C -- no --> E["logs an error and OMITS the field<br/>— does not raise, does not exit"]
    D --> F["build record_id · record_module<br/>· form_field_id · field_value"]
    F --> G["FormFieldValue.save_custom_fields"]
    E --> H(["later KeyError on the missing field name,<br/>deep inside the command"])

    classDef bad fill:#fde2e2,stroke:#a33,color:#000
    class E,H bad

regscale jcam check_custom_fields walks the same path ahead of time and reports
what is missing — it also exits 0 either way, so read its output rather than its
exit code.

8. Export flow — RegScale → JCAM, with drift detection

flowchart TD
    Start(["regscale jcam export_ssp / export_poams"]) --> Confirm{"--yes?"}
    Confirm -- no --> Prompt["click.confirm overwrite warning"]
    Prompt -->|abort| End([exit])
    Prompt -->|proceed| Filter
    Confirm -- yes --> Filter

    Filter["get_filtered_ssps — jcamFilter"] --> Branch{"export"}
    Branch --> FM["export_jcam_front_matter"]
    Branch --> AU["export_jcam_authorization"]
    Branch --> PM["push_jcam_poams"]

    subgraph Drift["per SSP"]
        direction TB
        Cur["retrieve_from_jcam — current JCAM state"]
        Des["build the desired payload from<br/>SecurityPlan / Issue"]
        Diff["detect_drift — compare key by key,<br/>ignoring keys JCAM has left empty"]
        Force{"--force?"}
        Skip(["record_drift and skip this item<br/>— the run continues"])
        Write["put_to_jcam / post_to_jcam"]
    end

    FM --> Cur
    AU --> Cur
    PM --> Cur
    Cur --> Des --> Diff
    Diff -- "no difference" --> NoOp(["nothing to write"])
    Diff -- "drift" --> Force
    Force -- no --> Skip
    Force -- yes --> Write
    Write --> JCAM[("JCAM")]

    classDef jcam fill:#fde2e2,stroke:#a33,color:#000
    class JCAM jcam

An empty value on the JCAM side is not treated as drift — writing into a blank field
is not a data-loss event. --force is what a first bulk load into a never-populated
JCAM instance needs, because everything there registers as drifted against defaults.

9. ODP parameter reconciliation — `import_parameters`

The only import that will not write what it cannot prove. Everything below EXACT
goes to a workbook for a human.

flowchart TD
    A(["regscale jcam import_parameters"]) --> B["fetch JCAM parameter values<br/>per system and control"]
    B --> N["normalize_jcam_parameters"]
    N --> CW{"--crosswalk-path given?"}
    CW -- yes --> LC["load_crosswalk — prior human decisions"]
    CW -- no --> REC
    LC --> REC["OdpReconciler — match against<br/>RegScale ControlParameter definitions"]
    REC --> T{"ConfidenceTier"}
    T -->|EXACT| DR{"--dry-run?"}
    T -->|STRONG| RQ
    T -->|WEAK| RQ
    T -->|AMBIGUOUS| RQ
    T -->|ORPHAN| RQ
    DR -- no --> AP["upsert_parameter — written to RegScale"]
    DR -- yes --> NOW["nothing written — reported only"]
    RQ["Review Queue"] --> WB["build_workbook →<br/>Mapping · Review Queue · Summary"]
    AP --> WB
    NOW --> WB
    WB --> OUT["xlsx in ~/Downloads<br/>(or --workbook-out)"]
    OUT --> HUM["a human edits the decisions"]
    HUM -->|"--learn-from &lt;workbook&gt;"| MG["merge_from_workbook → crosswalk"]
    MG -.->|"feeds the next run"| LC

    classDef ok fill:#e8f5e9,stroke:#2e7d32,color:#000
    class AP ok

10. Endpoint ↔ RegScale entity map

flowchart LR
    subgraph JCAMEP["JCAM — {jcamBasePath}/v1"]
        E1["/systems"]
        E2["/systems/{id}"]
        E3["/systems/{id}/securityauthorization"]
        E4["/systems/{id}/privacy · /sorn"]
        E5["/systems/{id}/continuityresponse · /continuitytest"]
        E6["/systems/{id}/status · /additionalstatus"]
        E7["/systems/{id}/infotypes"]
        E8["/systems/{id}/interconnections"]
        E9["/systems/{id}/FISMArollup"]
        E10["/systems/{id}/systempointsofcontact · /pocs"]
        E11["/systems/{id}/controls/{ctl} · /inheritedcontrols"]
        E12["/systems/{id}/controls/{ctl}/assessments"]
        E13["/systems/{id}/poams · /poams/{pid}/milestones · /controls"]
        E14["/systems/{id}/artifacts · /artifacts/{aid}"]
        E15["/systems/{id}/agencydefineddataitems"]
    end

    subgraph RSE["RegScale entities"]
        R1["SecurityPlan"]
        R2["FormFieldValue — custom fields"]
        R3["Authorization fields on the SSP"]
        R4["ControlImplementation + inheritance"]
        R5["Assessment / ControlTestResult"]
        R6["Issue (POA&M) + Milestones"]
        R7["File"]
        R8["User · SystemRole"]
        R9["ControlParameter"]
    end

    E1 --> R1
    E2 --> R1
    E2 --> R2
    E3 --> R3
    E4 --> R2
    E5 --> R2
    E6 --> R2
    E7 --> R2
    E8 --> R2
    E9 --> R1
    E10 --> R8
    E11 --> R4
    E11 --> R9
    E12 --> R5
    E13 --> R6
    E14 --> R7
    E15 --> R2
    E15 --> R1

/agencydefineddataitems splits: a few well-known attributes land on native
SecurityPlan columns (High Value Asset, Cloud System, Cloud Deployment Model,
Cloud Service Model), everything else on custom fields named by
jcamAgencyDefinedDataItems.

11. Run accounting — why exit 0 is not proof of a clean sync

flowchart TD
    R["JcamRunReport — thread-safe counters<br/>success · skipped · drift · failure"] --> D["failures deduped by (domain, message)<br/>with [xN] multipliers"]
    D --> L["log_summary — ERROR level when<br/>anything failed, INFO otherwise"]
    L --> F["finalize_run"]
    F --> Z(["process exits 0 regardless"])

    classDef warn fill:#fff4e5,stroke:#b26a00,color:#000
    class Z warn

finalize_run is deliberately visibility-only: exiting non-zero has automation blast
radius and diverges from the framework's exit-0 convention. Read the summary block
at the end of a run — an exit code of 0 with a per-domain failure line means part of
the sync did not happen.