Adverity data ingestion pipelines process CSV content that may contain headers, quoted commas, escaped quotes, and fields spanning multiple physical lines. Implement yield_csv_batches to parse the input and return rows grouped into batches without manually splitting on commas.
The function receives lines, an iterable of CSV text lines, batch_size, a positive integer, and has_header, a boolean. Return a list of batches. Each batch contains at most batch_size rows. If has_header is true, represent each data row as a dictionary mapping header names to string values. Otherwise, represent each row as a list of strings. Preserve the order of rows and fields, skip completely blank CSV records, and do not return the header as data.
Raise ValueError when batch_size is not positive or when a header contains duplicate names. An empty input, or an input containing only a header, returns an empty list.
def yield_csv_batches(lines, batch_size, has_header):