
Meta's lead ingestion pipeline for Facebook Lead Ads occasionally creates duplicate submission records for the same lead and form. Write a SQL query to identify the canonical record to keep for each duplicated lead and return only the duplicate rows that should be removed.
lead_id and form_id.submitted_at ascending, then ingested_at ascending, then submission_id ascending.lead_submissions| column | type | description |
|---|---|---|
| submission_id | INT | Unique submission row ID |
| lead_id | VARCHAR(20) | Lead identifier from Meta lead capture |
| form_id | VARCHAR(20) | Lead form identifier |
| campaign_id | INT | Associated ad campaign |
| submitted_at | TIMESTAMP | Time the lead was submitted |
| ingested_at | TIMESTAMP | Time the row landed in the warehouse |
| VARCHAR(100) | Lead email address | |
| phone | VARCHAR(20) | Lead phone number |
| source_surface | VARCHAR(30) | Meta surface where the lead originated |
campaigns| column | type | description |
|---|---|---|
| campaign_id | INT | Campaign identifier |
| advertiser_id | INT | Owning advertiser |
| campaign_name | VARCHAR(100) | Campaign name |
| status | VARCHAR(20) | Campaign status |
| start_date | DATE | Campaign start date |
advertisers| column | type | description |
|---|---|---|
| advertiser_id | INT | Advertiser identifier |
| advertiser_name | VARCHAR(100) | Advertiser name |
| vertical | VARCHAR(50) | Advertiser industry |
| region | VARCHAR(30) | Primary region |
Representative rows are shown in the provided dataset.
| submission_id | lead_id | form_id | campaign_name | advertiser_name | submitted_at | ingested_at | duplicate_rank |
|---|---|---|---|---|---|---|---|
| 102 | L1001 | F201 | Q1 SMB Lead Gen | Acme Fitness | 2024-03-01 09:00:00 | 2024-03-01 09:02:00 | 2 |
| 104 | L1002 | F201 | Q1 SMB Lead Gen | Acme Fitness | 2024-03-02 10:00:00 | 2024-03-02 10:06:00 | 2 |
| 109 | L1006 | F204 | Reengagement Retargeting | Nova Education | 2024-03-05 14:00:00 | 2024-03-05 14:05:00 | 2 |
| Column | Type | Description |
|---|---|---|
| submission_idPK | INT | Unique submission row ID |
| lead_id | VARCHAR(20) | Lead identifier from Meta lead capture |
| form_id | VARCHAR(20) | Lead form identifier |
| campaign_id | INT | Associated ad campaign |
| submitted_at | TIMESTAMP | Time the lead was submitted |
| ingested_at | TIMESTAMP | Time the row landed in the warehouse |
| VARCHAR(100) | Lead email address | |
| phone | VARCHAR(20) | Lead phone number |
| source_surface | VARCHAR(30) | Meta surface where the lead originated |
| Column | Type | Description |
|---|---|---|
| campaign_idPK | INT | Campaign identifier |
| advertiser_id | INT | Owning advertiser |
| campaign_name | VARCHAR(100) | Campaign name |
| status | VARCHAR(20) | Campaign status |
| start_date | DATE | Campaign start date |
| Column | Type | Description |
|---|---|---|
| advertiser_idPK | INT | Advertiser identifier |
| advertiser_name | VARCHAR(100) | Advertiser name |
| vertical | VARCHAR(50) | Advertiser industry |
| region | VARCHAR(30) | Primary region |
| campaign_id | advertiser_id | campaign_name | status | start_date |
|---|---|---|---|---|
| 14 | 1004 | Creators Webinar Leads | Paused | 2024-03-01 |
| 10 | 1001 | Q1 SMB Lead Gen | Active | 2024-02-15 |
| 12 | 1002 | Spring Travel Leads | Paused | 2024-03-01 |
| 11 | 1003 | Shops Merchant Signup | Active | 2024-02-20 |
| 13 | 1005 | Reengagement Retargeting | Active | 2024-02-25 |
| 15 | 1001 | Messenger Follow-up | Archived | 2024-01-10 |
| 16 | 1006 | Advantage+ Auto Leads | Active | 2024-03-05 |
| 17 | 1007 | Local Services Test | Draft | 2024-03-07 |
| 1 | 96 | Messenger Follow-up | Draft | 2024-02-16 |
| 2 | 29 | Seasonal Reactivation | Paused | 2024-02-19 |
| 3 | 45 | SMB Credit Offers | Active | 2024-02-21 |
| 4 | 18 | App Install Retargeting | Paused | 2024-03-05 |
| 5 | 82 | Local Services Test | Paused | 2024-02-24 |
| 6 | 70 | Advantage+ Auto Leads | Draft | 2024-01-10 |
| 7 | 95 | Lead Quality Experiment | Archived | 2024-02-20 |
| 8 | 56 | Fintech Prospecting | Active | 2024-02-16 |
| 9 | 36 | App Install Retargeting | Paused | 2024-01-14 |
| 10 | 10 | Retail Catalog Expansion | Draft | 2024-02-21 |
| 11 | 27 | Fintech Prospecting | Active | 2024-02-16 |
| 12 | 18 | Click-to-Messenger Leads | Draft | 2024-01-25 |
| 13 | 69 | Reengagement Retargeting | Archived | 2024-01-09 |
| 14 | 6 | Creator Monetization Push | Paused | 2024-01-21 |
| 15 | 21 | Retail Catalog Expansion | Active | 2024-03-07 |
| 16 | 36 | Travel Early Access | Paused | 2024-02-14 |
| 17 | 4 | Real Estate Buyer Leads | Active | 2024-02-28 |
| 18 | 98 | Lead Quality Experiment | Completed | 2024-01-09 |
| 19 | 43 | Creator Monetization Push | Active | 2024-01-27 |
| 20 | 15 | Lead Quality Experiment | Draft | 2024-03-04 |
| 21 | 36 | Healthcare Consult Leads | Archived | 2024-01-31 |
| 22 | 91 | App Install Retargeting | Active | 2024-01-20 |
| advertiser_id | advertiser_name | vertical | region |
|---|---|---|---|
| 1003 | ShopSphere | Retail | EMEA |
| 1001 | Acme Fitness | Health & Wellness | NA |
| 1005 | Nova Education | Education | APAC |
| 1002 | Bright Travel | Travel | EMEA |
| 1004 | CreatorHub | Media | |
| 1006 | Urban Eats | Food Delivery | LATAM |
| 1007 | HomeHero | Home Services | NA |
| 1008 | FinEdge | Finance | EMEA |
| 1 | SolarGrid | Energy | US |
| 2 | CareBridge | Automotive | NA |
| 3 | ZenWell | Fitness | LATAM |
| 4 | CoreBank | Health & Wellness | EMEA |
| 5 | NextStep Learning | Hospitality | Mexico |
| 6 | CraftLane | Media | Australia |
| 7 | ShopSphere | Health & Wellness | null |
| 8 | Skyline Realty | Finance | Mexico |
| 9 | NextStep Learning | Nonprofit | South Korea |
| 10 | FreshTable | Beauty | EMEA |
| 11 | Luma Beauty | Retail | MEA |
| 12 | PetNest | Healthcare | MEA |
| 13 | Pulse Health | Finance | Global |
| 14 | Pulse Health | Insurance | South Korea |
| 15 | GreenCart | Automotive | EMEA |
| 16 | BlueWave Media | E-commerce | South Korea |
| 17 | QuickCover | Finance | US |
| 18 | GreenCart | Professional Services | APAC |
| 19 | Luma Beauty | Education | Global |
| 20 | TrailWorks | Gaming | null |
| submission_id | lead_id | form_id | campaign_id | submitted_at | ingested_at | phone | source_surface | |
|---|---|---|---|---|---|---|---|---|
| 105 | L1003 | F202 | 12 | 2024-03-03 11:00:00 | 2024-03-03 11:01:00 | maria@bright.com | 555-0105 | |
| 101 | L1001 | F201 | 10 | 2024-03-01 09:00:00 | 2024-03-01 09:01:00 | alex@example.com | 555-0101 | |
| 110 | L1007 | F205 | 2024-03-06 08:00:00 | 2024-03-06 08:03:00 | sam@unknown.com | Messenger | ||
| 103 | L1002 | F201 | 10 | 2024-03-02 10:00:00 | 2024-03-02 10:05:00 | jordan@example.com | 555-0102 | |
| 108 | L1006 | F204 | 13 | 2024-03-05 14:00:00 | 2024-03-05 14:04:00 | lee@nova.com | ||
| 102 | L1001 | F201 | 10 | 2024-03-01 09:00:00 | 2024-03-01 09:02:00 | alex@example.com | 555-0101 | |
| 111 | L1008 | F206 | 14 | 2024-03-06 09:00:00 | 2024-03-06 09:02:00 | 555-0108 | ||
| 104 | L1002 | F201 | 10 | 2024-03-02 10:00:00 | 2024-03-02 10:06:00 | jordan@example.com | 555-0102 | |
| 106 | L1004 | F203 | 11 | 2024-03-04 12:00:00 | 2024-03-04 12:02:00 | nina@shop.com | 555-0106 | |
| 109 | L1006 | F204 | 13 | 2024-03-05 14:00:00 | 2024-03-05 14:05:00 | lee@nova.com | ||
| 107 | L1005 | F203 | 11 | 2024-03-04 12:30:00 | 2024-03-04 12:31:00 | omar@shop.com | 555-0107 | Audience Network |
| 112 | L1009 | F207 | 99 | 2024-03-07 15:00:00 | 2024-03-07 15:01:00 | taylor@missing.com | 555-0109 | |
| 1 | L1017 | F201 | 10 | 2024-03-08 13:08:58 | 2024-03-02 13:20:10 | omar@shop.com | 555-0116 | Facebook Stories |
| 2 | L1016 | F219 | 14 | 2024-03-06 00:25:13 | 2024-03-02 11:43:54 | lucas@homehero.com | 555-0104 | Reels |
| 3 | L1008 | F216 | 79 | 2024-03-04 18:40:24 | 2024-03-09 04:20:54 | omar@shop.com | 555-0122 | Instant Articles |
| 4 | L1006 | F205 | 8 | 2024-03-04 23:48:48 | 2024-02-27 12:10:02 | oliver@market.ai | 555-0102 | Instagram Stories |
| 5 | L1025 | F204 | 54 | 2024-02-27 09:53:15 | 2024-03-03 06:31:55 | mia@urban.com | 555-0124 | Shops |
| 6 | L1009 | F212 | 61 | 2024-02-27 10:26:45 | 2024-03-02 09:00:38 | taylor@missing.com | 555-0106 | Messenger |
| 7 | L1003 | F224 | 13 | 2024-03-04 02:53:15 | 2024-03-08 04:10:42 | ava@finedge.com | 555-0110 | |
| 8 | L1018 | F213 | 16 | 2024-03-02 15:13:20 | 2024-03-10 14:20:40 | null | 555-0113 | Facebook Stories |
| 9 | L1007 | F208 | 87 | 2024-03-05 02:29:48 | 2024-03-09 10:59:46 | alex@example.com | 555-0125 | Messenger |
| 10 | L1001 | F218 | 67 | 2024-03-09 12:54:23 | 2024-03-07 14:11:29 | jordan@example.com | 555-0117 | |
| 11 | L1014 | F218 | 52 | 2024-03-05 20:43:49 | 2024-03-02 07:57:01 | isabella@edu.org | 555-0122 | Profile |
| 12 | L1004 | F218 | 10 | 2024-03-06 23:41:28 | 2024-03-08 15:17:09 | benjamin@meta-partner.com | null | Events |
| 13 | L1026 | F202 | 62 | 2024-03-05 07:12:32 | 2024-03-01 19:24:26 | lucas@homehero.com | 555-0119 | |
| 14 | L1021 | F219 | 35 | 2024-03-06 05:28:51 | 2024-03-03 09:42:19 | null | 555-0117 | Shops |
| 15 | L1014 | F226 | 20 | 2024-02-28 11:58:14 | 2024-03-08 06:28:57 | daniel@travelco.com | 555-0120 | Marketplace |
| 16 | L1013 | F204 | 51 | 2024-03-07 06:53:44 | 2024-03-02 17:56:59 | priya@acme.com | null | Reels |
| submission_id | lead_id | form_id | campaign_name | advertiser_name | submitted_at | ingested_at | duplicate_rank |
|---|---|---|---|---|---|---|---|
| 102 | L1001 | F201 | Q1 SMB Lead Gen | Acme Fitness | 2024-03-01 09:00:00 | 2024-03-01 09:02:00 | 2 |
| 104 | L1002 | F201 | Q1 SMB Lead Gen | Acme Fitness | 2024-03-02 10:00:00 | 2024-03-02 10:06:00 | 2 |
| 109 | L1006 | F204 | Reengagement Retargeting | Nova Education | 2024-03-05 14:00:00 | 2024-03-05 14:05:00 | 2 |