Problem

You’re on an on-call rotation for a fintech event-ingestion pipeline that processes tens of millions of events per day. Downstream fraud and compliance systems deduplicate events by hashing a canonical string representation of each payload. Unfortunately, partners send payloads with inconsistent whitespace, optional null fields, and non-deterministic object key order.

Write a function canonicalize_payload(payload) that converts a JSON-like Python object into a deterministic string with the exact formatting rules below.

Input Types

payload is one of:

• dict[str, Any]
• list[Any]
• str, int, float, bool, or None

Canonicalization Rules

1. Dictionaries (objects)
   • Omit any key whose value is None.
   • Sort remaining keys lexicographically.
   • Render as {k1:v1,k2:v2} with no spaces.
   • Quote only the keys "id" and "tags" (i.e., output "id":... and "tags":...). All other keys are unquoted.
2. Lists (arrays)
   • Preserve element order.
   • Render as [e1,e2,...] with no spaces.
3. Strings
   • Trim leading/trailing whitespace.
   • Collapse any run of whitespace (spaces, tabs, newlines) into a single space.
   • Always wrap the normalized string in double quotes.
4. Scalars
   • None -> null
   • True/False -> true/false (lowercase)
   • Numbers render via str(x).

Examples

Example 1

• Input: {"id": " evt_1 ", "meta": {"b": 2, "a": 1}, "tags": [" vip", "vip ", "new"], "note": None}
• Output: {"id":"evt_1","meta":{a:1,b:2},"tags":["vip","vip","new"]}
• Explanation: Drop note, sort meta keys, normalize string whitespace, preserve list order, and quote only id and tags keys.

Example 2

• Input: [{"k": "a b"}, {"k": "a b"}, None, True]
• Output: [{k:"a b"},{k:"a b"},null,true]
• Explanation: Both strings normalize to "a b"; array order is preserved; None and True become null and true.

Notes

• If a dict becomes empty after dropping None values, it must render as {}.
• None inside a list is not dropped; it renders as null.