How far can I push a $7 server? - Importing CSV with Queues

In the previous experiment, I tried to import 5 million contacts synchronously on a $7 server in under 10 minutes.

After several implementations, indexes, bulk inserts, upserts and chunk-size experiments, I managed to get surprisingly close. But the final question was probably the most obvious one:

Should the user really be waiting for this import to finish?

Probably not.

So this time I'm going to move the work to queues, let the user continue immediately and see what actually changes.

Once the user doesn't have to wait for the request anymore, the 10-minute limit stops being a product requirement. I can accept the import, tell the user that it is being processed and notify them when it is done.

That sounds better already. But what new problems are we buying in exchange?

Case 1: Move everything to a queue

Let's start with the obvious option: take the exact same importer from the previous article and execute it inside a queue worker.

The implementation doesn't change. The only difference is that the import is now executed by a queue worker instead of during the request.

So what happens when the same work becomes asynchronous? I ran the asynchronous version three times, using the same 5 million contact dataset. The results below show the average of those three runs, compared with the synchronous baseline from the previous experiment.

The result is a little bit slower than the synchronous version. Moving the importer to a queue doesn't reduce the amount of work: the same CSV still needs to be parsed, the same database statements still need to be executed and the same vCPU is doing the work. But we did gain something important: the user doesn't have to wait anymore. Memory usage increased. What happened here?

The queue needs a worker running in the background waiting for jobs. On this server, an idle Laravel worker uses around 65 MB of memory. During the import, that process stays alive while the job is running, so moving the work to a queue also means keeping that additional process in memory.

This server only has 1 GB of RAM and normally uses around 60% of it just running the system and its services, so 65 MB isn't completely irrelevant here. If we want to add more workers later, memory is something we'll need to keep an eye on.

Now the question is: can we improve the processing time?

Case 2: Split that CSV

The import is asynchronous now, but one worker is still processing the whole file sequentially. If we want multiple workers to help, we need to split the work into independent jobs.

There are several ways we could do that:

• Load chunks into memory and pass them to jobs.
• Store chunks in the database first.
• Physically split the original CSV into smaller files.
• Give each job an offset into the original CSV.

All of them have one thing in common: at some point we need to walk through the CSV and decide which rows belong to each job. What is the cost of doing this?

My first implementation used fgetcsv() to traverse the file. I measured how much time it needed and found that traversing the whole file took about 3 minutes. We are talking about 30% of the total import time. That is a lot of time! fgetcsv() needs to parse the text line as a CSV, handling columns, delimiters and quoted values, and that takes time. For a small CSV file it is probably not a big deal, but for a large file, every millisecond counts.

In PHP, there is another function that maybe we can use. It is called fgets(), and it reads a line from the file without parsing it as CSV. Let's try it and see if it is worth it.

Just to put it into perspective, for a CSV with 100,000 rows we are talking about roughly 3.6 seconds with fgetcsv() versus 0.02 seconds with fgets(). Probably not something we would even care about. With 5 million rows, however, those 3.6 seconds become 3 minutes.

There is one problem with using fgets() here. It reads physical lines, not CSV records. In my dataset, every contact is in one line, so it works. But this is not something we can always assume. A valid CSV can have line breaks inside quoted fields, so one line doesn't necessarily mean one contact.

In a real import, we would need to validate this first or find the chunk boundaries in a CSV-aware way. That would add some of the work we just removed. For this experiment, I know that every contact is in one line, so I'm going to keep using fgets().

Now that traversing the file is cheap, let's go back to the options:

• Load chunks into memory: with only 1 GB of RAM, keeping large chunks in memory is not really an option.
• Store chunks in the database: the database is already doing most of the work during the import. Adding more writes just to store the chunks would put even more load on it.
• Physically split the CSV: this means reading the original file, writing new files to disk and then reading those files again when the jobs process them. That's additional CPU and I/O that we can avoid.
• Use offsets: each job can open the original CSV, quickly reach its starting point using fgets() and only parse the rows it actually needs with fgetcsv().

So, for me, offsets win.

About the implementation, each job receives an absolute offset and a row count, uses fgets() to reach its starting point and processes its assigned rows with fgetcsv(). The database logic remains the same: batches of 5,000 contacts with one SELECT, one INSERT and one UPSERT per batch.

<?php
$handle = fopen($this->scenario['csv'], 'r');

$rows = 0;

$file_chunk_size = 500000;
$offset = 0;

fgets($handle); // skip the header row

$jobs = array();

while(($row = fgets($handle)) !== false) {

    $rows++;

    if($rows % $file_chunk_size == 0){
        $jobs[] = new ProcessContactImportChunkJob(
            $contact_import_run->id,
            $offset,
            $file_chunk_size
        );
        $offset += $file_chunk_size;
    }

}

$remaining = $rows % $file_chunk_size;

if ($remaining > 0) {
    $jobs[] = new ProcessContactImportChunkJob(
        $contact_import_run->id,
        $offset,
        $remaining
    );
}


fclose($handle);

Bus::batch($jobs)
    ->then(function (Batch $batch) use ($contact_import_run_id) {
        // logic when all jobs in the batch have been processed
    })
    ->dispatch();

Example of a worker processing a chunk:

<?php

$chunk_size = 5000;
$contacts_to_check = array();

$created = 0;
$updated = 0;

$handle = fopen($this->scenario['csv'], 'r');
$headers = fgetcsv($handle);

$rows_to_skip = $this->offset;

//skip lines
while ($rows_to_skip > 0 && fgets($handle) !== false) {
    $rows_to_skip--;
}

$rows = 0;

while($rows < $this->file_chunk_size && ($row = fgetcsv($handle)) !== false) {
    
    //process batch

    
    $rows++;
}

if(count($contacts_to_check) > 0){
    //process remaining contacts in the last batch
}


fclose($handle);

Now the import is split into independent jobs, so let's try processing them with two workers on the original $7 server. These are the results:

The result isn't much better. What is going on? If the two workers are processing simultaneously, shouldn't it be faster? Yes, but in this case, we have one problem... we just have one vCPU. The two workers are now competing for the same single vCPU. And one vCPU can't really execute two CPU-bound tasks in parallel.

This is the difference between concurrency and parallelism. We have concurrency: both workers are making progress because the vCPU switches between them. But we don't have parallelism: with only one vCPU, both workers can't actually execute at the same time.
We changed the architecture so it can use more vCPUs, but the server still only has one. Can upgrading the server actually help now?

Case 3: Upgrading the server

In the previous article, upgrading the server from $7 to $21 wasn't particularly helpful. The importer was sequential, so buying a second vCPU didn't give the code much opportunity to use it. Now we have a different architecture, so let's see if this time it is worth paying for.

Server specifications:

I tested both configurations on both servers, three runs each:

• One async job, one worker.
• 500k chunks, two workers.

Same dataset. Same database logic. Same chunk sizes. The only variables are worker count and available CPU capacity.

One worker

With one worker, the larger server doesn't change much.

With one worker, the work is still being processed sequentially. The second vCPU is mostly sitting there waiting for something to do.

Two workers

Now let's actually give that second vCPU some work.

This is the point where the extra vCPU can finally be used by the workload. Now our two workers can actually execute jobs in parallel.

The conclusion here isn't simply that a bigger server is faster. In the previous article, the importer was sequential and couldn't make much use of the second vCPU. After splitting the work into independent jobs, the architecture can actually use that extra vCPU. If your architecture can't use the extra vCPU, buying a bigger server won't help much.

So, what is the price of using queues?

Moving the import to queues improved the user experience and made parallel processing possible, but it also introduced a completely new set of decisions.

Memory

Workers are long-lived processes and they consume memory even while idle. The more workers we keep alive, the less headroom the rest of the application has. Long-running workers may also retain allocated memory, so periodic restarts are a normal part of operating them.

Worker management

Something needs to keep the workers alive and restart them when necessary. That can be Supervisor, systemd or another process manager. It isn't complicated, but it is one more moving part.

UX is better, but more complex

The request returns immediately, but now the application needs a way to tell the user when the import is finished. In a browser that might mean polling, WebSockets, email notifications or some combination of them.

Failures

Not every failure means the same thing. A timeout, server restart, killed process or temporary lack of resources can often be retried. A deterministic error while inserting a specific batch may fail again every single time.

Retries and idempotency

Retrying a job means executing it again. The important part is making that safe. If a 500k-row chunk dies after 250k rows, running it again should not duplicate contacts or leave the import in an inconsistent state.

Partial failures

If one batch fails, the product needs a policy. Abort the whole import? Ask the user to upload it again? Continue with the rest and report something like “created X, updated Y, rejected Z”? There is no universal answer.

Job coordination

Once the import is split into multiple jobs, we also need to know when the whole import is finished. In Laravel, job batches make this easier: we can group the jobs into a batch and run some logic when all of them have completed. Without something like this, we would need to keep track of the individual jobs ourselves.

Observability

Once the work happens in the background, we need to know what state it is in: whether it is pending, running, completed or failed, how many rows were processed and which jobs are still alive.

Throughput vs CPU headroom

In the benchmark, the CPU can stay close to full utilization when multiple workers are active. That may be fine for a dedicated import server, but it leaves very little room for web requests, database queries or other background jobs running on the same machine.

The important part is understanding that async processing isn't free. It trades a simpler synchronous flow for better UX, decoupling and the ability to run work concurrently.

Conclusion

In Case 1, moving the same importer to a queue didn't improve performance.

The work became asynchronous, but it was still one worker doing the same work sequentially. What improved was the user experience: the request could return immediately and the import could continue in the background.

In Case 2, splitting the CSV made the workload parallelizable, but two workers on one vCPU didn't help. They were both competing for the same vCPU and the total import time stayed roughly the same.

In Case 3, the extra vCPU finally became useful.

With two vCPUs and two workers, two chunks could actually run at the same time.

Queues didn't make the import faster by themselves. What they changed was how the work could be executed. Once the import was split into independent jobs and the server had enough CPU to run them in parallel, adding capacity finally made a meaningful difference.

Of course, the tradeoffs are real. Workers consume memory, need supervision, failures need policies, retries need idempotency, background work needs observability and maximum throughput can leave almost no CPU headroom for the rest of the application.

Is async worth it?

For this import, yes, mainly because the user no longer has to wait.

The performance improvement only appeared once the workload was split and the server actually had enough CPU to execute that work in parallel.

Which leaves another question.

What happens when two users start a huge import at the same time?

The vCPU has declined to comment.