PHP Generators Streaming Large Data Sets

By Kokil Thapa | Last reviewed: September 2026

Exporting 500,000 order rows crashes your PHP script at 512 MB. The fix is not always a bigger server. PHP generators streaming large data sets process one record at a time and keep memory flat. I've hit this on production Laravel apps during nightly CSV exports and legal document batch jobs. This guide shows how generators work, when they beat arrays, and how to wire them to PDO, Eloquent, and HTTP responses on PHP 8.3 through 8.5.

What are PHP generators and how do they stream large data sets?

A generator is a function that returns an iterator via yield. Each call to next() runs the function until the next yield. Then execution pauses. State is preserved between yields. You never materialise the full collection.

Compare that to array_map() or fetchAll(). Those load every row into RAM first. On a custom enterprise application with two million audit log rows, that difference ends the job before it starts.

Basic generator syntax

The simplest pattern reads a large file line by line. No full file load required.

<?php
declare(strict_types=1);

function readLargeFile(string $path): Generator
{
    $handle = fopen($path, 'rb');
    if ($handle === false) {
        throw new RuntimeException("Cannot open {$path}");
    }

    try {
        while (($line = fgets($handle)) !== false) {
            yield rtrim($line, "\r\n");
        }
    } finally {
        fclose($handle);
    }
}

foreach (readLargeFile('/var/exports/orders.csv') as $line) {
    /* process one line */
}

PHP treats any function containing yield as a generator. The return type hint Generator documents intent. Use finally so file handles close even when the loop breaks early.

Generator delegation with yield from

yield from delegates to another iterable or generator. That lets you compose pipelines without nesting loops.

function filterActiveUsers(Generator $users): Generator
{
    foreach ($users as $user) {
        if ($user['status'] === 'active') {
            yield $user;
        }
    }
}

function streamUsers(): Generator
{
    yield from filterActiveUsers(fetchUsersFromDb());
}

The official PHP manual documents generators and the iterator protocol in detail. See the PHP generator language reference for return values and send() semantics.

When should you use PHP generators instead of loading arrays?

Generators shine when data volume is unknown or too large for RAM. They are a poor fit when you need random access, repeated full scans, or small fixed lists.

On the Nepal Gift Card platform, order history exports use generators. Loading every gift-card transaction into an array would spike memory during peak sales season.

Rule of thumb: if the dataset can exceed available PHP memory minus overhead, stream it. Check your PHP-FPM memory limits before assuming 512 MB is enough.

• Streaming CSV or JSON downloads to the browser
• Nightly reconciliation files from payment gateways
• Import pipelines that validate row-by-row
• Log aggregation across rotated files
• Read-only reporting where SQL aggregation is not possible

How do you stream database results with PDO and Laravel?

Generators alone do not help if PDO buffers the entire result set server-side. You must disable buffering for true streaming from MySQL 9.7 or MariaDB 12.3.

PDO with MYSQL_ATTR_USE_BUFFERED_QUERY disabled

<?php
function streamOrders(PDO $pdo): Generator
{
    $pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);

    $stmt = $pdo->query(
        'SELECT id, total, created_at FROM orders WHERE status = "paid"'
    );

    try {
        while ($row = $stmt->fetch(PDO::FETCH_ASSOC)) {
            yield $row;
        }
    } finally {
        $stmt->closeCursor();
        $pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, true);
    }
}

Always restore buffered mode after streaming. Leaving it off breaks later queries on the same connection. Always close the cursor in finally.

MySQL documents cursor and result-set behaviour in the MySQL 9.7 cursor reference. Unbuffered mode holds the read lock until the cursor closes.

Laravel Eloquent cursor() and lazy()

Laravel 12 and 13 expose cursor() for generator-style iteration over Eloquent models. It uses PHP generators internally and fetches one row at a time.

use App\Models\Order;

function exportPaidOrders(): Generator
{
    foreach (Order::query()
        ->where('status', 'paid')
        ->select(['id', 'total', 'created_at'])
        ->cursor() as $order) {
        yield [
            'id' => $order->id,
            'total' => $order->total,
            'date' => $order->created_at->toDateString(),
        ];
    }
}

lazy() fetches in chunks of 1000 by default. Use it when each model triggers accessors or small relation lookups. Use cursor() for bare exports.

On booking systems like Adventure Third Pole Trek, I've streamed trek booking rows for accounting exports. Selecting only needed columns avoids hydrating full models.

Watch for N+1 queries inside the loop. Eager loading does not work the same way with cursors. Join the data in SQL or accept denormalised export queries.

Chunking for writes vs generators for reads

chunkById() suits batch updates. Generators suit read-only streaming. Do not mix them blindly.

1. Use cursor() or a PDO generator for read/export paths.
2. Use chunkById() when updating 50,000 rows in batches of 500.
3. Queue a job per chunk when work exceeds HTTP timeout.
4. Log progress every N rows for observability.

How do you stream CSV exports and HTTP responses with generators?

The real win is sending data to the browser while it is still being read from MySQL. The user gets bytes immediately. Memory stays low on both sides.

Symfony and plain PHP streamed response

<?php
function streamCsvResponse(Generator $rows, array $headers): void
{
    header('Content-Type: text/csv; charset=UTF-8');
    header('Content-Disposition: attachment; filename="export.csv"');
    header('Cache-Control: no-cache');

    $out = fopen('php://output', 'wb');
    fputcsv($out, $headers);

    foreach ($rows as $row) {
        fputcsv($out, $row);
    }

    fclose($out);
}

Disable output buffering before streaming. In PHP-FPM, call ob_end_flush() in a loop until no buffer remains. Laravel's StreamedResponse wraps this pattern cleanly.

use Symfony\Component\HttpFoundation\StreamedResponse;

return new StreamedResponse(function () {
    $out = fopen('php://output', 'wb');
    foreach (exportPaidOrders() as $row) {
        fputcsv($out, $row);
    }
    fclose($out);
}, 200, [
    'Content-Type' => 'text/csv',
    'Content-Disposition' => 'attachment; filename="orders.csv"',
]);

Long streams can exceed nginx or Apache proxy timeouts. Raise fastcgi_read_timeout or move very large exports to a queued job that writes to object storage. Then email a signed download link.

Validate exported JSON with a JSON formatter during development. Broken escaping shows up fast in small samples.

Combining generators with API pagination

External APIs paginate results. Wrap pagination in a generator so callers still see a flat iterable.

function streamApiPages(callable $fetchPage): Generator
{
    $page = 1;

    do {
        $response = $fetchPage($page);
        foreach ($response['data'] as $item) {
            yield $item;
        }
        $page++;
    } while ($response['meta']['has_more'] === true);
}

Add exponential backoff on rate-limit responses. I've used this pattern for payment reconciliation on production apps. Each page yields rows without storing prior pages.

What mistakes break PHP generator memory savings in production?

Generators do not magically fix bad code. Several habits reintroduce memory spikes or lock tables for minutes.

Mistake 1: collecting into an array inside the loop

/* BAD — defeats the generator */
$all = [];
foreach (streamOrders($pdo) as $row) {
    $all[] = transform($row);
}
return $all;

Process or write each row immediately. If you need aggregation, use SQL GROUP BY or a rolling counter variable.

Mistake 2: keeping unbuffered connections open too long

An open unbuffered read blocks writes to InnoDB tables in some configurations. Stream during off-peak windows. Or replicate to a read replica for heavy exports.

Mistake 3: ignoring time and memory limits

Generators fix memory. They do not fix CPU or wall-clock time. Set set_time_limit(0) only in CLI exports. For HTTP, use queues.

Run load tests with k6 against export endpoints before launch. A flat memory graph means nothing if the request times out at 60 seconds.

Mistake 4: serialising generator output

Generators are not serialisable. Do not pass them to Redis or cache drivers. Materialise to a file path or job payload instead.

Static analysis catches some misuse. Add PHPStan level 9 to CI so return types stay honest.

Monitoring and testing

Log peak memory with memory_get_peak_usage(true) at the end of CLI exports. Compare against the same export using fetchAll(). The difference should be orders of magnitude.

For WooCommerce-scale catalogues, see WooCommerce speed optimisation for large catalogs. WordPress often loads full post arrays where a custom SQL stream would suffice.

On grocery platforms like Quick And Easy Nepalese Grocery, delivery-zone order exports run as Artisan commands. Generators plus unbuffered PDO keep nightly cron inside 128 MB.

Key Takeaways

• Use yield and yield from to iterate large files, API pages, and query results without building arrays.
• Disable PDO buffered queries when streaming MySQL; restore the setting and close cursors in finally.
• Prefer Laravel cursor() for lean exports and lazy() when you need chunked model hydration.
• Stream CSV to php://output with output buffering disabled; queue jobs for exports over proxy timeouts.
• Never collect yielded rows into a second array — process, write, or aggregate inline.
• Load-test export endpoints and log peak memory so flat RAM claims hold under production traffic.

People Also Ask

Do PHP generators work with PHP 8.5?

Yes. Generators have been stable since PHP 5.5. PHP 8.5 adds no breaking changes to yield semantics. Typed generator return values and strict types work as expected. All examples here run on PHP 8.3 through 8.5 with Composer 2.10.

How much memory do PHP generators save?

Memory stays near O(1) relative to row count when you stream correctly. A one-million-row export might use 30–80 MB with a generator versus 800 MB or more with fetchAll(). Exact numbers depend on row width and PHP overhead.

Can Laravel queues use generators directly?

No. Queue jobs serialise their payload. Pass a query scope or file path to the job instead. Let the worker build the generator inside handle(). That pattern keeps Redis payloads small and jobs retry-safe.

Are generators faster than arrays?

Not always. Generators reduce memory and often start output sooner. Per-row overhead can be slightly higher than bulk array iteration. Choose generators when memory or streaming latency matters, not raw microsecond speed on small datasets.

Ship exports that survive production traffic

PHP generators streaming large data sets are the right tool when arrays would blow memory or delay downloads. Pair generators with unbuffered PDO or Laravel cursors, stream directly to HTTP or disk, and avoid collecting rows mid-pipeline. That is the pattern I rely on for exports on Laravel apps serving real clients in Nepal and abroad.

Need help refactoring a failing export or designing a streaming API? Review our testing and optimisation services or API development work. For full-stack delivery including deployment and LEMP stack setup, see web development services. About me covers 15+ years of production PHP work. When exports outgrow single-server streaming, test data pipelines and Redis-backed queues scale the workload. Ongoing support catches memory regressions before your clients do. Contact us to audit your export path.