PyYAML Duplicate Keys: Test Merge Overrides Before Rejecting Them

PyYAML Duplicate Keys: Test Merge Overrides Before Rejecting Them

Todd Mercer

Coral and teal folds overlap while a violet crescent stays distinct, separating intentional overrides from accidental duplicates.

PyYAML safe_load does not reject repeated mapping keys in the tested 6.0.3 environment. But adding a duplicate check after merge expansion can reject a configuration that intentionally overrides a default.

The practical decision is not simply “strict or permissive YAML.” Decide separately whether explicit mapping keys must be unique and whether your consumer permits << merges. Check the original mappings before constructing dictionaries, then assert the merged values.

Aident tested 15 small synthetic configurations against four policies using Python 3.13.14 and PyYAML 6.0.3. The complete fixture below passed all 15 acceptance-vector assertions and eight additional result assertions. These are configuration-policy results, not a security audit or a benchmark of every YAML parser.

The merge override you should not mistake for a duplicate

Consider a service overriding its default port:

base: &base { host: local, port: 8080 }
service: { <<

Under the tested allow-merge policy, service becomes {"host": "local", "port": 9090}. The service mapping has one explicit port key. The inherited port is not an accidental repeated explicit key.

A flattened-key uniqueness check sees both ports and rejects this fixture. That is a policy mismatch if your consumer allows defaults to be overridden.

By contrast, this source contains two explicit keys and is rejected by both checked policies:

service: {port: 8080, port: 9090}

The upstream duplicate-key discussion identifies this interaction between duplicate checking and merges. The YAML 1.1 merge-key draft describes inherited entries yielding to explicit keys, and earlier mappings winning within a merge sequence. It is a merge-extension contract, not a promise that every YAML 1.2 consumer supports it.

What the four policies accepted

“Accepted” means the loader returned without raising for that synthetic input. It does not mean the application schema, required fields, value types, or business rules passed.

Configuration case

Default safe_load

Unique after flattening

Check original keys, allow merge

Check original keys, refuse merge

Ordinary string-key mapping

Accept

Accept

Accept

Accept

Duplicate root key

Accept

Reject

Reject

Reject

Duplicate nested key

Accept

Reject

Reject

Reject

Same decoded name through a Unicode escape

Accept

Reject

Reject

Reject

Merge default plus explicit override

Accept

Reject

Accept

Reject

Two merge sources sharing a key

Accept

Reject

Accept

Reject

Merge sources plus explicit override

Accept

Reject

Accept

Reject

Duplicate inside a merge source

Accept

Reject

Reject

Reject

Two separate merge directives

Accept

Accept

Reject

Reject

Quoted literal "<<" key

Accept

Accept

Accept

Accept

Boolean/integer mapping-key collision

Accept

Reject

Reject

Reject

Sequence used as a mapping key

Reject

Reject

Reject

Reject

Recursive alias

Accept

Accept

Reject

Reject

Unsupported Python object tag

Reject

Reject

Reject

Reject

Scalar used as a merge source

Reject

Reject

Reject

Reject

The “unique after flattening” loader is deliberately a diagnostic counterexample, not a recommended validator. It both rejects intended overrides and accepts the repeated merge-directive fixture because its two sources have disjoint resulting keys.

The two checked policies deliberately require string mapping keys, at most one merge directive per mapping, and no recursive aliases. Those are application choices, not universal YAML validity rules.

Assert precedence, not just successful parsing

For the tested version:

a: &a { host: first }
b: &b { host: second }
service: { <<

The expected service host is first. Adding an explicit host: explicit to the service changes the expected result to explicit. The fixture asserts both results and uses a multi-character key.

Issue 610 explains why merge-sequence precedence differs from repeated-key last-wins behavior. A later upstream main-branch report and proposed fix are reasons to rerun this test when changing versions. This article does not reproduce that branch or claim the report affects the installed 6.0.3 release.

Version numbers, loader classes, input, and recovered values belong in the acceptance record. “It parsed” is too weak to detect a changed default.

Run the complete acceptance fixture

Use a disposable Python environment with PyYAML==6.0.3. The tested run used the pure-Python yaml.SafeLoader, not CSafeLoader. For managed execution, paste the fixture into Code Interpreter Tools, inspect the current Action inputs and spend approval, and wait for its terminal output.

The fixture operates on in-memory synthetic strings. It does not read your workspace, credentials, or customer configuration. The first pass examines the original node graph; the second uses safe_load to construct the result. Duplicate evidence has not already been erased into a dictionary.

import json
import sys
import yaml
from yaml.constructor import ConstructorError
from yaml.nodes import MappingNode, ScalarNode, SequenceNode

assert yaml.__version__ == "6.0.3"
STR = "tag:yaml.org,2002:str"
MERGE = "tag:yaml.org,2002:merge"


def load_checked(text, allow_merge=True):
    if len(text.encode("utf-8")) > 16384:
        raise ValueError("fixture input too large")
    root = yaml.compose(text, Loader=yaml.SafeLoader)
    visited, active = set(), set()

    def check(node, depth=0):
        if depth > 20 or id(node) in active:
            raise ValueError("depth or recursive alias outside policy")
        if id(node) in visited:
            return
        visited.add(id(node))
        if len(visited) > 200:
            raise ValueError("node count outside policy")
        active.add(id(node))
        if isinstance(node, MappingNode):
            seen, merge_seen = set(), False
            for key, value in node.value:
                if key.tag == MERGE:
                    if not allow_merge or merge_seen:
                        raise ValueError("merge directive outside policy")
                    merge_seen = True
                else:
                    if not isinstance(key, ScalarNode) or key.tag != STR:
                        raise ValueError("only string mapping keys allowed")
                    if key.value in seen:
                        raise ValueError("duplicate explicit key: " + key.value)
                    seen.add(key.value)
                check(value, depth + 1)
        elif isinstance(node, SequenceNode):
            for child in node.value:
                check(child, depth + 1)
        active.remove(id(node))

    if root is not None:
        check(root)
    return yaml.safe_load(text)


class FlattenedUniqueLoader(yaml.SafeLoader):
    def construct_mapping(self, node, deep=False):
        self.flatten_mapping(node)
        seen = set()
        for key_node, _ in node.value:
            key = self.construct_object(key_node, deep=deep)
            if key in seen:
                raise ConstructorError(None, None, "duplicate flattened key", key_node.start_mark)
            seen.add(key)
        return super().construct_mapping(node, deep=deep)


cases = [
    ("ordinary", "service: {host: local, port: 8080}", [1, 1, 1, 1]),
    ("root_duplicate", "host: first\nhost: second", [1, 0, 0, 0]),
    ("nested_duplicate", "service: {host: first, host: second}", [1, 0, 0, 0]),
    ("decoded_name_duplicate", 'host: first\n"\\u0068ost": second', [1, 0, 0, 0]),
    ("merge_override", "base: &base {host: local, port: 8080}\nservice: {<<: *base, port: 9090}", [1, 0, 1, 0]),
    ("merge_sequence", "a: &a {host: first}\nb: &b {host: second}\nservice: {<<: [*a, *b]}", [1, 0, 1, 0]),
    ("merge_sequence_override", "a: &a {host: first}\nb: &b {host: second}\nservice: {<<: [*a, *b], host: explicit}", [1, 0, 1, 0]),
    ("duplicate_in_source", "base: &base {host: first, host: second}\nservice: {<<: *base}", [1, 0, 0, 0]),
    ("repeated_merge_directive", "a: &a {host: first}\nb: &b {port: 9090}\nservice: {<<: *a, <<: *b}", [1, 1, 0, 0]),
    ("quoted_merge_text", '"<<": literal\nhost: local', [1, 1, 1, 1]),
    ("typed_key_collision", "true: first\n1: second", [1, 0, 0, 0]),
    ("complex_key", "? [a, b]\n: value", [0, 0, 0, 0]),
    ("recursive_alias", "service: &service {self: *service}", [1, 1, 0, 0]),
    ("unsafe_python_tag", "service: !!python/object:builtins.object {}", [0, 0, 0, 0]),
    ("invalid_merge_value", "service: {<<: scalar}", [0, 0, 0, 0]),
]
policies = {
    "safe_load": yaml.safe_load,
    "flattened_unique": lambda s: yaml.load(s, Loader=FlattenedUniqueLoader),
    "checked_allow_merge": load_checked,
    "checked_no_merge": lambda s: load_checked(s, allow_merge=False),
}
rows = []
for name, text, expected in cases:
    observed = []
    reasons = []
    for policy in policies.values():
        try:
            policy(text)
            observed.append(1)
            reasons.append(None)
        except (ValueError, TypeError, yaml.YAMLError) as error:
            observed.append(0)
            reasons.append(type(error).__name__ + ": " + str(error).split("\n")[0])
    assert observed == expected, (name, observed, expected)
    rows.append({"case": name, "accepted": observed, "reasons": reasons})

assert load_checked(cases[4][1])["service"] == {"host": "local", "port": 9090}
assert load_checked(cases[5][1])["service"]["host"] == "first"
assert load_checked(cases[6][1])["service"]["host"] == "explicit"
assert load_checked(cases[9][1])["<<"] == "literal"
assert yaml.safe_load(cases[10][1]) == {True: "second"}
assert yaml.safe_load("flag: on")["flag"] is True
assert load_checked("flag: on")["flag"] is True
assert load_checked("defaults: &d {host: local}\nservice: {<<: *d}")["service"] == {"host": "local"}
print(json.dumps({"python": sys.version, "pyyaml": yaml.__version__, "policies": list(policies), "cases": rows, "assertions": 23, "status": "PASS"}, indent=2))

Expected output: status is PASS, pyyaml is 6.0.3, all 15 rows match the table, and assertions is 23. The final eight assertions independently check overrides, first-source precedence, a literal merge-looking key, typed-key collapse, the retained Boolean value resolver, and a simple merge.

If the version assertion fails, do not delete it to make the run green. Record the new version, rerun the same cases, and review any changed vector or recovered value.

What this check does not guarantee

The helper accepts a deliberately narrow configuration contract. It leaves PyYAML's value resolution intact: flag: on still becomes a Boolean in the tested environment. Requiring string keys does not require string values.

The true and 1 key fixture collapses to one Python dictionary entry under default loading. The checked policies avoid that class of collision by refusing non-string keys. They do not implement a general resolver for all possible YAML key equivalences.

The input-size, graph-depth, node-count, and cycle checks are fixture boundaries, not a denial-of-service defense. Composition happens before the graph checks. The test does not exercise parser resource exhaustion, huge alias graphs, every tag, all Unicode normalization, multiple-document streams, custom constructors, or alternate loaders. Do not deploy this demonstration as a hardened public upload validator.

PyYAML's documentation distinguishes safe loading from constructors that can create arbitrary Python objects. Staying with SafeLoader avoids those constructors; it does not establish duplicate-key, schema, resource, or consumer correctness by itself.

For real configuration ingestion, define a schema after construction: required keys, allowed value types, ranges, unknown-field policy, and the exact service values you expect. Keep the original input and policy decision available for review, with secrets removed.

Put the test at the consumer boundary

Choose an execution boundary before supplying a real configuration. Code interpreter versus AI-agent sandbox explains why a successful remote fixture does not imply access to your local application or its configuration.

If the resulting configuration becomes a JSON tool payload, apply the downstream decoder's own contract. The Python JSON acceptance test covers duplicate names and number policies that YAML merge rules do not solve. These are separate checks, not interchangeable parser settings.

Before allowing an agent to act on a configuration, keep three artifacts together: the redacted source, the chosen duplicate/merge policy, and the asserted resolved values. When the loader version or consumer policy changes, rerun the entire fixture rather than only the happy-path input.

Run the YAML configuration acceptance fixture with synthetic inputs first. A passing result establishes the tested policy behavior; connecting an integration, executing against real systems, and approving external writes are separate decisions.

About the author

Illustrated editorial avatar of Todd Mercer

Todd Mercer

Todd Mercer is a senior engineer with more than 10 years of experience working on social media platforms and large-scale ordering systems. He graduated from the University of Illinois Urbana-Champaign in computer science. At Aident, he covers technical issues, developer tools, and technology topics.

Home

Home

Home

Integrations

Integrations

Integrations

Vault

Vault

Vault

Audit

Audit

Audit

Arana Grande

Arana Grande

Arana Grande

Free

Free

Free

30-day audit summary

30-day audit summary

30-day audit summary

Daily action-call volume and the latest receipts from the Loadout audit trail.

Daily action-call volume and the latest receipts from the Loadout audit trail.

Daily action-call volume and the latest receipts from the Loadout audit trail.

View Audit

View Audit

View Audit

Loadout usage

Loadout usage

Loadout usage

617 action calls in the last 30 days

617 action calls in the last 30 days

617 action calls in the last 30 days

May 19 - Jun 17

May 19 - Jun 17

May 19 - Jun 17

10 active days

10 active days

10 active days

Less

Less

Less

More

More

More

Recent activity

Recent activity

Recent activity

Latest action-call receipts from connected agents

Latest action-call receipts from connected agents

Latest action-call receipts from connected agents

Apr 23, 09:23 AM

Apr 23, 09:23 AM

Apr 23, 09:23 AM

Shopify

Shopify

Shopify

Creates Or Updates An Asset For A Theme

Creates Or Updates An Asset For A Theme

Creates Or Updates An Asset For A Theme

Success

Success

Success

Apr 23, 09:21 AM

Apr 23, 09:21 AM

Apr 23, 09:21 AM

Shopify

Shopify

Shopify

Update Products Param Product Id

Update Products Param Product Id

Update Products Param Product Id

Success

Success

Success

Apr 23, 08:53 AM

Apr 23, 08:53 AM

Apr 23, 08:53 AM

Shopify

Shopify

Shopify

Update Products Param Product Id

Update Products Param Product Id

Update Products Param Product Id

Failed

Failed

Failed

Apr 22, 22:13 PM

Apr 22, 22:13 PM

Apr 22, 22:13 PM

Shopify

Shopify

Shopify

Create Product Image

Create Product Image

Create Product Image

Success

Success

Success

Apr 22, 22:12 PM

Apr 22, 22:12 PM

Apr 22, 22:12 PM

Shopify

Shopify

Shopify

Create Product Image

Create Product Image

Create Product Image

Success

Success

Success

Connected integration coverage

Connected integration coverage

Connected integration coverage

162

162

162

of 753 accessible connected

of 753 accessible connected

of 753 accessible connected

Callable actions

Callable actions

Callable actions

1,126

1,126

1,126

Vault credentials

Vault credentials

Vault credentials

8

8

8

Explore what's possible

Explore what's possible

Explore what's possible

See all Integrations

See all Integrations

See all Integrations

Google Ads

Google Ads

Google Ads

All available Goolge Ads tools via...

All available Goolge Ads tools via...

All available Goolge Ads tools via...

X (twitter)

X (twitter)

X (twitter)

All available X tools via...

All available X tools via...

All available X tools via...

Github

Github

Github

All available Github tools via...

All available Github tools via...

All available Github tools via...

Notion

Notion

Notion

All available Notion tools via...

All available Notion tools via...

All available Notion tools via...

Slack

Slack

Slack

All available Slack tools via...

All available Slack tools via...

All available Slack tools via...

Firecrawl

Firecrawl

Firecrawl

All available Firecrawl tools via...

All available Firecrawl tools via...

All available Firecrawl tools via...

753 integrations are available for loadouts.

753 integrations are available for loadouts.

753 integrations are available for loadouts.

The one tool

for every tool

your agent needs.

Give any AI agent real capabilities in seconds. Connect 27,000+ tools once, skip the setup headache, and let your agents execute.

Try Aident Loadout

Empower your Codex or OpenClaws to get real jobs done. Connect 27,000+ tools in one prompt, and let your agents deliver real results.

Try Aident Loadout

Empower your Codex or OpenClaws to get real jobs done. Connect 27,000+ tools in one prompt, and let your agents deliver real results.

Try Aident Loadout

Empower your Codex or OpenClaws to get real jobs done. Connect 27,000+ tools in one prompt, and let your agents deliver real results.