How to Build a Real-Time CMDB from Azure and AWS APIs

Most enterprise CMDBs are wrong. Not slightly wrong — wrong in ways that matter. Resources that exist aren't in it. Resources that were deleted are still in it. Tags are stale. Relationships between compute, network, and identity are missing entirely. And the ServiceNow discovery agents that are supposed to keep it current run on a 24-hour cycle, use WMI and SNMP protocols designed for on-premises infrastructure, and have no native understanding of cloud resource models.

The result is a CMDB that IT operations doesn't trust, security can't use for blast radius analysis, and FinOps ignores in favor of their own spreadsheets. The system of record for your infrastructure isn't a record of anything.

ServiceNow does offer certified Service Graph Connectors for both Azure and AWS. They work, and if you're already deep in the ServiceNow ecosystem they're worth evaluating. But they have real limitations: they run on a polling schedule, they map resources into ServiceNow's CI class model which loses cloud-specific properties, they don't normalize across clouds, and you have no control over the ingestion logic. For enterprises running multi-cloud at scale, building your own pipeline from the source APIs gives you accuracy, latency, and flexibility that the connectors don't.

Both Azure and AWS maintain a continuously updated, queryable inventory of every resource in your environment. The CMDB problem is a pipeline problem, not a discovery problem.

Why Discovery-Based CMDBs Fail in the Cloud

Traditional CMDB discovery works by scanning networks, connecting to endpoints, and interrogating running systems. That model has three fundamental problems in cloud environments:

Ephemeral resources don't survive the scan cycle. An auto-scaling group spins up 40 VMs to handle a batch job, processes it in 90 minutes, and terminates. A 24-hour discovery cycle never sees them. But those VMs had IAM roles, accessed S3 buckets, wrote to databases, and generated CloudTrail events. They existed. Your CMDB has no record of them.

Cloud resource models don't map to CI classes. A ServiceNow “Server” CI was designed for a physical or virtual machine with a fixed IP, a hostname, and an OS. An Azure Container App is none of those things. It's a managed runtime with a revision model, ingress rules, Dapr sidecar configuration, environment variables referencing Key Vault secrets, and a managed identity with role assignments. Forcing that into a Server CI loses 90% of the operationally relevant data.

Relationships are inferred, not authoritative. Discovery agents guess relationships by correlating IP addresses, hostnames, and port scans. Cloud providers know the relationships exactly — because they enforce them. Azure knows that a VM is in a subnet, that subnet is in a VNet, that VNet is peered to a hub, and that hub has a route table with a specific UDR pointing to an NVA. That relationship graph is in the API. Discovery agents reconstruct a shadow of it.

Azure Resource Graph: The Authoritative Inventory

Azure Resource Graph is a query service that indexes every resource across all subscriptions in your tenant. It supports a full KQL (Kusto Query Language) interface and can return resource properties, tags, relationships, and change history. Importantly, Resource Graph data is eventually consistent — Microsoft documents it as having a short indexing latency, and change records are available in under five minutes. It is not strongly consistent. Design your pipeline accordingly: use it for near-real-time sync, not as a transactional source of truth.

The key table is Resources, which contains every ARM resource across your subscriptions. A basic inventory query:

Resources
| project
    id,
    name,
    type,
    location,
    resourceGroup,
    subscriptionId,
    tags,
    properties,
    identity,
    sku,
    kind,
    tenantId
| order by type asc

Results are paginated at 1000 records per page. For large tenants you'll page through using the $skipToken in the REST response. The API allows 15 requests per 3 seconds per subscription — for tenants with many subscriptions, parallelize at the subscription level with a token bucket to stay under the limit.

For virtual machines, extract the properties that matter for your CI class:

Resources
| where type == "microsoft.compute/virtualmachines"
| extend
    vmSize = properties.hardwareProfile.vmSize,
    osType = properties.storageProfile.osDisk.osType,
    osDiskSizeGB = properties.storageProfile.osDisk.diskSizeGB,
    provisioningState = properties.provisioningState,
    powerState = properties.extended.instanceView.powerState.code,
    nicIds = properties.networkProfile.networkInterfaces,
    subnetId = properties.networkProfile.networkInterfaces[0].properties.subnet.id,
    managedIdentityType = identity.type,
    principalId = identity.principalId
| project
    id, name, location, resourceGroup, subscriptionId,
    vmSize, osType, osDiskSizeGB, provisioningState,
    powerState, nicIds, subnetId,
    managedIdentityType, principalId, tags

For change tracking, Resource Graph exposes the resourcechanges table (lowercase in queries). It has a 14-day retention window — critical for pipeline design. If your incremental sync falls behind by more than 14 days you will have gaps and must fall back to a full sync. The change record schema uses nested properties:

resourcechanges
| extend
    changeTime = todatetime(properties.changeAttributes.timestamp),
    changeType = tostring(properties.changeType),
    changedBy = tostring(properties.changeAttributes.changedBy),
    clientType = tostring(properties.changeAttributes.clientType),
    targetResourceId = tostring(properties.targetResourceId),
    targetResourceType = tostring(properties.targetResourceType)
| where changeTime > ago(1h)
| project
    targetResourceId, targetResourceType,
    changeType, changedBy, clientType, changeTime,
    properties.changes
| order by changeTime desc

Note: for changeType == "Delete", the changesCount is 0 because the resource is gone. For changeType == "Create", changesCount is also 0 by design — logging every property on creation would be too noisy. Use changeType to drive your CMDB upsert/retire logic, not the changes diff.

For network topology, reconstruct the graph from NIC resources — they are the join point between compute and network:

Resources
| where type == "microsoft.network/networkinterfaces"
| extend
    vmId = tostring(properties.virtualMachine.id),
    subnetId = tostring(properties.ipConfigurations[0].properties.subnet.id),
    privateIp = tostring(properties.ipConfigurations[0].properties.privateIPAddress),
    nsgId = tostring(properties.networkSecurityGroup.id),
    publicIpId = tostring(properties.ipConfigurations[0].properties.publicIPAddress.id)
| project id, name, vmId, subnetId, privateIp, nsgId, publicIpId, resourceGroup

For role assignments, the AuthorizationResources table covers RBAC. Join it against role definitions to get human-readable role names:

AuthorizationResources
| where type == "microsoft.authorization/roleassignments"
| extend
    principalId = tostring(properties.principalId),
    principalType = tostring(properties.principalType),
    roleDefinitionId = tostring(properties.roleDefinitionId),
    scope = tostring(properties.scope)
| join kind=leftouter (
    AuthorizationResources
    | where type == "microsoft.authorization/roledefinitions"
    | project roleDefinitionId = id, roleName = tostring(properties.roleName)
) on roleDefinitionId
| project principalId, principalType, roleName, scope

AWS Config: The Equivalent on the AWS Side

AWS Config records the configuration state of every resource in your accounts and streams change events to EventBridge. For multi-account environments, the standard 2026 pattern is a delegated administrator account with an Organization-level aggregator — not a manually created aggregator per account. You enable Config in all member accounts via AWS Organizations, designate a central security or operations account as the delegated admin, and the aggregator automatically picks up all current and future accounts.

# Enable delegated admin for Config (run from management account)
aws organizations register-delegated-administrator \
  --account-id 123456789012 \
  --service-principal config.amazonaws.com

# Create org-wide aggregator (run from delegated admin account)
aws configservice put-configuration-aggregator \
  --configuration-aggregator-name org-aggregator \
  --organization-aggregation-source '{
    "RoleArn": "arn:aws:iam::123456789012:role/ConfigAggregatorRole",
    "AllAwsRegions": true
  }'

One cost consideration before you build: AWS Config charges $0.003 per configuration item recorded and $0.0012 per rule evaluation. At 100K resources across 20 accounts that's real money. Scope your recorder to the resource types you actually need in the CMDB — don't record everything by default.

For bulk inventory, use the aggregator advanced query interface. The Limit parameter maxes at 100 records per page — always set it explicitly and paginate via NextToken:

import boto3

def query_all_resources(resource_type: str, aggregator: str) -> list:
    config = boto3.client("config", region_name="us-east-1")
    results = []
    next_token = None

    while True:
        kwargs = {
            "ConfigurationAggregatorName": aggregator,
            "Limit": 100,
            "Expression": f"""
                SELECT
                    resourceId,
                    resourceName,
                    resourceType,
                    accountId,
                    awsRegion,
                    availabilityZone,
                    configuration,
                    tags,
                    configurationItemCaptureTime
                WHERE
                    resourceType = '{resource_type}'
            """
        }
        if next_token:
            kwargs["NextToken"] = next_token

        response = config.select_aggregate_resource_config(**kwargs)
        results.extend(response.get("Results", []))
        next_token = response.get("NextToken")
        if not next_token:
            break

    return results

Note what's not in that SELECT: relationships. The advanced query SQL interface does not support querying the relationships field — it's not a queryable property in the aggregator SQL syntax. To get relationships you need to call batch_get_resource_config directly, which returns the full ConfigurationItem including the relationships array. That API is limited to 100 resources per call:

def get_relationships(resource_keys: list, region: str) -> dict:
    # resource_keys: [{"resourceType": "AWS::EC2::Instance", "resourceId": "i-xxx"}]
    # max 100 per call
    config = boto3.client("config", region_name=region)
    response = config.batch_get_resource_config(resourceKeys=resource_keys)

    relationships_map = {}
    for item in response.get("baseConfigurationItems", []):
        resource_id = item["resourceId"]
        # relationships field: list of {relationshipName, resourceType, resourceId, resourceName}
        relationships_map[resource_id] = item.get("relationships", [])

    return relationships_map

# Example relationships entry for an EC2 instance:
# {
#   "relationshipName": "Is contained in Vpc",
#   "resourceType": "AWS::EC2::VPC",
#   "resourceId": "vpc-0abc123",
#   "resourceName": ""
# }

The field is relationshipName, not name. This matters when you're parsing the response to build your CMDB relationship edges.

For change events, subscribe to EventBridge from the delegated admin account using an org-wide event bus policy. Route Config change notifications to SQS for durable processing:

# EventBridge rule — captures Config change notifications across all accounts
{
  "source": ["aws.config"],
  "detail-type": ["Config Configuration Item Change"],
  "detail": {
    "messageType": ["ConfigurationItemChangeNotification"],
    "configurationItem": {
      "resourceType": [
        "AWS::EC2::Instance",
        "AWS::RDS::DBInstance",
        "AWS::EKS::Cluster",
        "AWS::Lambda::Function",
        "AWS::S3::Bucket",
        "AWS::IAM::Role"
      ]
    }
  }
}

The Ingestion Pipeline Architecture

Two modes: full sync (initial load and daily reconciliation) and incremental sync (event-driven). Both feed the same normalization layer before writing to the CMDB.

The normalization layer is where most implementations go wrong. Define a canonical CI schema first — a cloud-agnostic representation of each resource type — and write two translators: ARM JSON to canonical, AWS Config JSON to canonical. The CMDB adapter only speaks canonical. This is what makes the pipeline extensible to GCP or OCI without rewriting the CMDB integration.

The Canonical CI Schema

@dataclass
class CanonicalCI:
    # Identity
    canonical_id: str          # sha256(cloud + account_id + resource_id)
    cloud_resource_id: str     # ARM resource ID or ARN
    cloud_provider: str        # "azure" | "aws" | "gcp"
    account_id: str            # subscription ID or AWS account ID
    region: str

    # Classification
    ci_class: str              # "compute.vm" | "network.subnet" | ...
    resource_type: str         # raw cloud type
    name: str

    # State
    state: str                 # "running" | "stopped" | "terminated" | "provisioning"
    last_seen: datetime
    last_changed: datetime
    change_source: str         # identity that made the last change

    # Ownership (derived from tags — enforce tag policy first)
    tags: dict[str, str]
    environment: str
    owner: str
    cost_center: str
    application: str

    # Relationships (resolved to canonical_ids)
    parent_id: str | None      # resource group / AWS account
    network_id: str | None     # VNet / VPC
    subnet_id: str | None
    identity_id: str | None    # managed identity / instance profile

    # Raw
    raw_config: dict

The canonical_id is deterministic — same resource always produces the same ID. This enables idempotent upserts: run the full sync as many times as you want and it converges to the correct state without duplicates.

Writing Back to ServiceNow

The write path uses the Import Set API for bulk operations — it supports transform maps that handle the canonical-to-ServiceNow CI class mapping and batch-processes records rather than making one API call per CI. Use your canonical_id stored in a custom field (e.g. u_cloud_canonical_id) as the reconciliation key. Never use sys_id as your external key — it's internal to ServiceNow and breaks on instance migrations.

import requests

def upsert_ci(ci: CanonicalCI, snow_instance: str, auth: tuple):
    table_map = {
        "compute.vm": "cmdb_ci_vm_instance",
        "compute.kubernetes": "cmdb_ci_kubernetes_cluster",
        "compute.function": "cmdb_ci_cloud_function",
        "network.vnet": "cmdb_ci_network",
        "database.sql": "cmdb_ci_database",
    }
    table = table_map.get(ci.ci_class, "cmdb_ci_cloud_service_account")

    payload = {
        "u_cloud_canonical_id": ci.canonical_id,
        "name": ci.name,
        "u_cloud_provider": ci.cloud_provider,
        "u_cloud_account": ci.account_id,
        "u_cloud_region": ci.region,
        "u_cloud_resource_id": ci.cloud_resource_id,
        "operational_status": "1" if ci.state == "running" else "2",
        "u_environment": ci.environment,
        "u_owner": ci.owner,
        "u_cost_center": ci.cost_center,
        "u_last_cloud_change": ci.last_changed.isoformat(),
        "u_change_source": ci.change_source,
    }

    url = f"https://{snow_instance}.service-now.com/api/now/table/{table}"
    existing = requests.get(
        url,
        params={"sysparm_query": f"u_cloud_canonical_id={ci.canonical_id}", "sysparm_limit": 1},
        auth=auth
    ).json().get("result", [])

    if existing:
        requests.patch(f"{url}/{existing[0]['sys_id']}", json=payload, auth=auth)
    else:
        requests.post(url, json=payload, auth=auth)

For relationships, write to cmdb_rel_ci after all CIs are upserted. Relationship types that matter: “Hosted on” (VM → host), “Contained by” (subnet → VNet), “Depends on” (app → database), “Managed by” (resource → identity).

Operational Considerations

What Breaks When You Skip This

• Incident response takes 3x longer. When a security event fires, the first question is “what else does this resource connect to?” If the CMDB doesn't have accurate relationships, the analyst builds the blast radius manually from cloud console queries. That's 45 minutes of work that should be a 30-second CMDB lookup.
• Change management approvals are rubber stamps. If the CMDB doesn't reflect current state, change advisory boards can't assess impact. They approve changes based on stale data and find out about the actual dependencies when something breaks in production.
• Compliance audits become manual evidence collection. SOC 2, ISO 27001, and FedRAMP all require evidence that you know what's in your environment and who has access to it. If the CMDB isn't authoritative, every audit cycle is a manual inventory exercise. At enterprise scale, that's weeks of engineering time per audit.
• FinOps can't allocate costs accurately. Cost allocation requires knowing which resources belong to which application, team, and cost center. If the CMDB doesn't have that mapping — because tags are missing or stale — cloud spend gets allocated to “untagged” and nobody owns it.

The cloud providers have already solved the hard part. Azure Resource Graph and AWS Config maintain a continuously updated, queryable, relationship-aware inventory of everything in your environment. Build the pipeline once, and your CMDB stays current automatically — no agents, no scan windows, no 30% miss rate.