AEP Dataset, Schema, and Dataflow Troubleshooting

When AEP ingestion fails, troubleshoot the path in order: source payload, dataflow run, mapping, XDM schema, dataset, and downstream service. Do not start by rebuilding the schema or repeatedly rerunning the full load. One representative record and the first actionable error usually identify the broken contract faster.

The key is to separate three questions: did data reach Adobe Experience Platform, did it conform to the target schema, and did the service you care about consume it? A green source connection answers only the first part of that chain.

Capture one failing run

Open the dataflow monitoring details and record the run ID, start time, source, target dataset, records received, records ingested, records failed, and error message. Use those identifiers when searching logs or raising a support case. A screenshot without a run ID is difficult to correlate later.

Check whether the failure affects every record or only a subset. A total failure often points to credentials, file access, mapping configuration, or an incompatible payload. Partial failures more often indicate malformed values, missing required fields, inconsistent date formats, or arrays and objects arriving at the wrong shape.

For batch ingestion, inspect the associated batch status and error diagnostics. For streaming ingestion, use a controlled test payload and retain the request and response identifiers. Never test with unredacted production personal data when a synthetic record can reproduce the issue.

Validate the source shape before the mapping

Compare the actual payload with the source fields configured in the dataflow. Confirm case-sensitive names, nesting, delimiters, encoding, headers, and whether a value is an object, array, number, boolean, or string. A timestamp that looks readable to a person may still fail if it does not use an accepted date-time representation.

Sample records from different source conditions, not only the clean first row. Include null values, empty arrays, late-arriving attributes, and the longest realistic strings. If a CSV export changes columns between regions or business units, stabilize that source contract before adding mapping exceptions in AEP.

Inspect mapping and XDM compatibility

Trace each important source field to its target XDM path. Look first at identities, timestamps, event type, required fields, enums, and custom field groups. A mapping can be syntactically saved while still sending semantically wrong data, such as an account ID into a person identity namespace.

Confirm that the target path exists on the schema attached to the dataset and that the data type matches. An object cannot be mapped safely into a scalar string, and a local date should not be treated as an absolute event timestamp without a timezone rule. The XDM schema design guide explains how to model field groups, identities, and record grain before ingestion.

Avoid changing a production schema merely to accept a bad source value. Normalize the source or mapping where appropriate. If the business model truly changed, assess every dataset, query, audience, destination, and profile dependency before introducing a new schema version.

Verify the dataset contract

A dataset is bound to a schema. Confirm the dataflow targets the intended dataset and sandbox, especially when names are similar across development, test, and production. Then inspect recent batches or query a small sample through Query Service to prove where the records landed.

If the use case requires Real-Time Customer Profile, verify that both schema and dataset are enabled for Profile and that the schema contains a valid identity. Enabling Profile is not retroactive in every operational sense: confirm the timing, ingestion state, and whether existing data needs to be reingested under the approved plan.

Do not enable every dataset for Profile as a troubleshooting shortcut. Reference data, high-volume events without a profile use case, and incorrectly modeled entities can increase profile volume or create unsafe identity relationships.

Check the downstream boundary

Data in a dataset does not guarantee an updated profile or audience. Profile ingestion, identity processing, merge policies, audience evaluation, and destination activation are separate boundaries. Prove the dataset first, then follow the record using a known identity and timestamp.

If the record exists but no profile appears, continue with the profile ingestion troubleshooting guide. If profiles are correct but counts differ, use the audience population mismatch guide and confirm evaluation method and policy.

Make the fix repeatable

After correcting the issue, rerun a small controlled batch or event and record the expected result at each boundary. Add source contract checks for required columns, types, timestamps, and identity formats before data reaches AEP. Promote mappings and schema changes through sandboxes using versioned deployment records rather than recreating them by hand.

Global teams should also document where data is collected and processed. EU and Germany deployments may require stricter minimization and consent review, while teams in the UK, US, Canada, Australia, UAE, Singapore, and India can have different contractual and regulatory constraints. The engineering response is not separate country schemas by default; it is one governed model with regional controls where the legal basis and use case require them.

The RTCDP course connects schemas, ingestion, identity, Profile, and activation into one operating model. For help diagnosing a persistent ingestion path, contact us.