Processing data with Rust

In data processing, Rust has emerged as a compelling alternative to traditional ETL languages like Python or Java. Its unique combination of performance, safety, and modern features makes it an attractive choice for handling large-scale data streams and complex computational tasks. Libraries like Polars make Rust a good option for building data processing pipelines. With its growing ecosystem of libraries and frameworks, Rust is primed to play a significant role in shaping the future of data-intensive applications.

As a toy project, I explored how to use Rust to transform AWS CloudFront logs into Parquet format. This need is now obsolete as of February 2025. AWS has released a new logging stack that writes Parquet files directly to S3. There is no need to have a converter going forward. However, for the already existing files, a converter might be still useful.

AWS Cloudfront Logs

Amazon CloudFront is a content delivery network (CDN) service offered by Amazon Web Services (AWS). CloudFront is designed to deliver content to users by caching it at edge locations around the world. This reduces the distance that the data must travel, and making the comunication faster. CloudFront has a feature to save the access logs to a S3 bucket using some formats, including TSV (tab-separated values).

This looks like this:

Jan 12:58 E2F28FYJOwT1P9.2024-01-23-11.84ac498a.gz
Jan 13:03 E2F28FYJOwT1P9.2024-01-23-11.c34e4413.gz
Jan 13:08 E2F28FYJOwT1P9.2024-01-23-12.db61b94a.gz
Jan 13:18 E2F28FYJOwT1P9.2024-01-23-12.46529b24.gz
Jan 13:28 E2F28FYJOwT1P9.2024-01-23-12.d77399a4.gz
Jan 13:33 E2F28FYJOwT1P9.2024-01-23-12.6f1c7b5e.gz
Jan 13:58 E2F28FYJOwT1P9.2024-01-23-12.a5e6ddca.gz
Jan 14:03 E2F28FYJOwT1P9.2024-01-23-12.4d5e6dbc.gz
Jan 14:18 E2F28FYJOwT1P9.2024-01-23-13.7125fd04.gz
Jan 14:38 E2F28FYJOwT1P9.2024-01-23-13.7d3dbdf0.gz

Looking into a log file we can find the following (breaking the long lines ino shorter for better readability).

❯ gzcat gz/E3F26FYJOTTOP6.2024-01-23-12.4d5e6dbc.gz | tr '\t' ' '
#Version: 1.0
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem 
 sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type 
 x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for 
 ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version fle-status 
 fle-encrypted-fields c-port time-to-first-byte x-edge-detailed-result-type 
 sc-content-type sc-content-len sc-range-start sc-range-end

2024-01-23 12:59:15 LAX53-P1 1897 100.100.100.100 GET d197w9j7zcw6og.cloudfront.net 
/ai/riding-hood/ 200 https://dev.l1x.be/ai/riding-hood/ 
Mozilla/5.0%20(Macintosh;%20Intel%20Mac%20OS%20X%2010_15_7)%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko)%20Chrome/114.0.0.0%20Safari/537.36 
- - RefreshHit UKrRsY4BzxJsshXB71YE7R2zxpoB3npsKy39TyIRKQJFw_DusKph4t== 
dev.l1x.be https 330 0.301 
- TLSv1.3 TLS_AES_128_GCM_SHA256 RefreshHit HTTP/1.1 - - 
48634 0.301 RefreshHit text/html - - -

It would be great to have a single file per day in a format, that supports applying summary statistics on the dataset. It is simple to add more data as a new day, with an ETL job that runs daily. This job might run hourly in high traffic environments like e-commerce sites for example.

Using Parquet

After working with these file formats for a long time, I believe Parquet is an excellent option for long-term storage of data. It is a columnar storage format that is becoming increasingly popular in the data engineering world due to its numerous advantages.

Here are some of the key reasons why using Parquet is a good idea for data engineering:

• Reduced Storage Requirements:

Parquet allows for more efficient data compression, resulting in significantly smaller file sizes compared to row-based formats like CSV. This is particularly beneficial for storing large datasets, as it can significantly reduce storage costs and the amount of disk space required. Combining this with compression, especially Zstandard (ZSTD), yields excellent results.

• Faster Query Performance Compared to Text Data Formats:

Parquet enables faster query execution compared to text data formats, especially for complex analytical queries that involve aggregations or filtering on specific columns. Since Parquet only reads the required data (usually the header or specific columns), it minimizes disk I/O and improves processing speed. This makes it well-suited for large-scale data analysis and data warehousing applications.

Using the Rust Parquet lib might be a bit more complex for a simple use case like this one. Luckily we have Polars, an open-source library written in Rust (also has a Python interface) for data analysis and manipulation. It is well designed to be fast and efficient and has a LazyFrame for larger data sets.

Creating a workflow

Reading the TSV file (actually reading and unzipping) is easy with Rust. One assumption I have is that the entire dataset fits in memory. In the future, I’d like to improve this by exploring lazy evaluation with iterators. For now, we can have a few hundred megabytes worth of TSV in memory. I believe it’s more important to understand when O(n) memory usage is acceptable rather than prematurely optimizing for stream-based processing.

Get a list of files with a glob

Rust offers two main ways (and possibly more that I’m unaware of) to work with files:

• glob::glob
• std::fs

For flexible patterns with wildcards, opt for glob. It efficiently iterates over matching files, letting you perform actions like printing paths or collecting them into a vector. If fine-grained control over individual entries is needed, and globbing patterns are simpler, consider std::fs. It allows filtering directory entries based on criteria like file type or extension, giving you more granular control over the processing tasks. Both approaches handle errors and offer customization options. I opted for the glob create.

The files have date in the name (for example: E2F28FYJOwT1P9.2024-01-23-11.84ac498a.gz) and we can create a simple glob representing a single day in the dataset.

use glob::{glob, Paths, PatternError};
fn get_files(date: &str) -> Result<Paths, PatternError> {
    glob(&format!("gz/*.{}*.gz", &date))
}

The first approach is to simple iterate over the glob, ignore the errors for now and collect the file content in an accumulator (hence making this O(n) memory).

for path in paths {
    if path.is_err() {
        error!("Could not process path with error: {:?}", path);
        continue;
    }
    let path: PathBuf = path.unwrap();
    info!("Processing path: {:?}", path);
    // panic if cannot unzip file
    let tsv_maybe: Result<String, Error> = uncompress_gzip(path.to_str().unwrap());
    // the unzipped content is a potentially multiline TSV (tab separated values)
    match tsv_maybe {
        Ok(tsv) => {
            let entries: Vec<Vec<String>> = split_tsv(tsv);
            acc.extend(entries)
        }
        Err(e) => error!("{:?}", e),
    }
}

Unzip and parse

The next phase is unzipping the files in memory and creating a Vec for each line. Assemling the lines together into a Vec<Vec> is simple and this is exactly what

fn split_tsv(tsv: String) -> Vec<Vec<String>> {
    //This is a bit unreadable
    // - splitting the TSV to lines
    // - removing lines starting with # or being empty
    // - split the lines by tab
    // - convert &str to String so it can be returned (maybe this is not necessary?)

    tsv.split('\n')
        .filter(|l| !l.starts_with('#'))
        .filter(|l| !l.is_empty())
        .map(|l| {
            l.split('\t')
                .map(|s| s.to_string())
                .collect::<Vec<String>>()
        })
        .collect()
}

This function produces the following:

[
  ["2024-01-23", "12:14:37", "LAX53-P1", ...],
  ["2024-01-23", "12:56:19", "SOF50-C1", ... ]
]

After having a 2D vector like the one above, we are almost ready to write the dataset to its permanent storage, a local folder in this case. There is a small problem with this dataset, though. The API that Polars has for creating a dataframe does not match the vector we got from processing TSVs. Polars expects a dataset that looks like this:

let s1 = Series::new("Fruit", &["Apple", "Apple", "Pear"]);
let s2 = Series::new("Color", &["Red", "Yellow", "Green"]);

let df: PolarsResult<DataFrame> = DataFrame::new(vec![s1, s2]);

As you can see the new dataframe has a vector of Series. Each vector contains the same kind of data, for example, fruit names or colors. We can simply transpose the 2D vector we have to produce something similar.

fn transpose<T>(v: Vec<Vec<T>>) -> Vec<Vec<T>> {
    // Transposing a 2D matrix
    // https://stackoverflow.com/questions/64498617/how-to-transpose-a-vector-of-vectors-in-rust

    assert!(!v.is_empty());
    let len = v[0].len();
    let mut iters: Vec<_> = v.into_iter().map(|n| n.into_iter()).collect();
    (0..len)
        .map(|_| {
            iters
                .iter_mut()
                .map(|n| n.next().unwrap())
                .collect::<Vec<T>>()
        })
        .collect()
}

I was not able to determine which SO answer was better just by reading the source codes. I decided that I would like to understand if there is a significant performance difference between these.

use criterion::{black_box, criterion_group, criterion_main, Criterion};

fn transpose1<T>(original: Vec<Vec<T>>) -> Vec<Vec<T>> {
    assert!(!original.is_empty());
    let mut transposed = (0..original[0].len()).map(|_| vec![]).collect::<Vec<_>>();

    for original_row in original {
        for (item, transposed_row) in original_row.into_iter().zip(&mut transposed) {
            transposed_row.push(item);
        }
    }

    transposed
}

fn transpose2<T>(v: Vec<Vec<T>>) -> Vec<Vec<T>> {
    assert!(!v.is_empty());
    let len = v[0].len();
    let mut iters: Vec<_> = v.into_iter().map(|n| n.into_iter()).collect();
    (0..len)
        .map(|_| {
            iters
                .iter_mut()
                .map(|n| n.next().unwrap())
                .collect::<Vec<T>>()
        })
        .collect()
}

fn criterion_benchmark(crit: &mut Criterion) {
    crit.bench_function("transpose1", |b| {
        b.iter(|| {
            let a: Vec<u64> = (0..1000).map(|v| v + 1000).collect();
            let b: Vec<u64> = (0..1000).map(|v| v + 1000).collect();
            let c: Vec<u64> = (0..1000).map(|v| v + 1000).collect();

            let vx: Vec<Vec<u64>> = vec![a, b, c];

            transpose1(black_box(vx))
        })
    });

    crit.bench_function("transpose2", |b| {
        b.iter(|| {
            let a: Vec<u64> = (0..1000).map(|v| v + 1000).collect();
            let b: Vec<u64> = (0..1000).map(|v| v + 1000).collect();
            let c: Vec<u64> = (0..1000).map(|v| v + 1000).collect();

            let vx: Vec<Vec<u64>> = vec![a, b, c];

            transpose2(black_box(vx))
        })
    });
}

criterion_group!(benches, criterion_benchmark);
criterion_main!(benches);

Invoking the test with cargo bench is easy and it produces an HTML output that is not that hard to understand. The two functions are very close to each other in terms of performance and I picked the slightly faster one.

❯ cargo bench
    Finished bench [optimized] target(s) in 0.13s
     Running unittests src/main.rs (target/release/deps/cf_logs-5058ea9e2adfad2d)

running 1 test
test tests::exploration_df ... ignored

test result: ok. 0 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running benches/bench.rs (target/release/deps/bench-199f64d850bf6279)
Gnuplot not found, using plotters backend
transpose1              time:   [34.802 µs 34.944 µs 35.087 µs]
                        change: [-0.2763% +0.0465% +0.3828%] (p = 0.79 > 0.05)
                        No change in performance detected.
Found 6 outliers among 100 measurements (6.00%)
  5 (5.00%) high mild
  1 (1.00%) high severe

transpose2              time:   [27.966 µs 28.068 µs 28.174 µs]
                        change: [-0.0055% +0.3648% +0.6945%] (p = 0.04 < 0.05)
                        Change within noise threshold.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

Based on this I opted for the transpose2 function.

As we have this out of the way time to create the function that produces the data that we can feed to Polars.

use polars::{prelude::NamedFrom, series::Series};

pub(crate) fn get_columns(transposed: Vec<Vec<String>>) -> Vec<Series> {
    // this creates a vec of Series from a 2D vec

    vec![
        Series::new("date", &transposed[0]),                // date
        Series::new("time", &transposed[1]),                // time
        Series::new("x-edge-location", &transposed[2]),     // x-edge-location
        Series::new("sc-bytes", &transposed[3]),            // sc-bytes
        Series::new("c-ip", &transposed[4]),                // c-ip
        Series::new("cs-method", &transposed[5]),           // cs-method
        Series::new("cs(Host)", &transposed[6]),            // cs(Host)
        Series::new("cs-uri-stem", &transposed[7]),         // cs-uri-stem
        Series::new("sc-status", &transposed[8]),           // sc-status
        Series::new("cs(Referer)", &transposed[9]),         // cs(Referer)
        Series::new("cs(User-Agent)", &transposed[10]),     // cs(User-Agent)
        Series::new("cs-uri-query", &transposed[11]),       // cs-uri-query
        Series::new("cs(Cookie)", &transposed[12]),         // cs(Cookie)
        Series::new("x-edge-result-type", &transposed[13]), //  x-edge-result-type
        Series::new("x-edge-request-id", &transposed[14]),  // x-edge-request-id
        Series::new("x-host-header", &transposed[15]),      // x-host-header
        Series::new("cs-protocol", &transposed[16]),        // cs-protocol
        Series::new("cs-bytes", &transposed[17]),           // cs-bytes
        Series::new("time-taken", &transposed[18]),         // time-taken
        Series::new("x-forwarded-for", &transposed[19]),    // x-forwarded-for
        Series::new("ssl-protocol", &transposed[20]),       // ssl-protocol
        Series::new("ssl-cipher", &transposed[21]),         // ssl-cipher
        Series::new("x-edge-response-result-type", &transposed[22]), // x-edge-response-result-type
        Series::new("cs-protocol-version", &transposed[23]), // cs-protocol-version
        Series::new("fle-status", &transposed[24]),         // fle-status
        Series::new("fle-encrypted-fields", &transposed[25]), // fle-encrypted-fields
        Series::new("c-port", &transposed[26]),             // c-port
        Series::new("time-to-first-byte", &transposed[27]), // time-to-first-byte
        Series::new("x-edge-detailed-result-type", &transposed[28]), // x-edge-detailed-result-type
        Series::new("sc-content-type", &transposed[29]),    // sc-content-type
        Series::new("sc-content-len", &transposed[30]),     // sc-content-len
        Series::new("sc-range-start", &transposed[31]),     // sc-range-start
        Series::new("sc-range-end", &transposed[32]),       // sc-range-end
    ]
}

There is only one step is remaining in the project: writing the Parquet files to disk.

let mut df: DataFrame = DataFrame::new(columns).unwrap();
let file_name: String = format!("par/{}.zstd.parq", date);
let mut file: File = File::create(file_name).unwrap();
Ok(ParquetWriter::new(&mut file).finish(&mut df)?)

This function writes the DataFrame to a Parquet file compressed with Zstandard.

Closing

Rust is a powerful language for data engineering with modern features and great libraries. By combining Rust and Polars (and the underlying great libraries), we can efficiently process and analyze datasets like AWS CloudFront logs. Give Rust a try for your next data engineering project—you won’t be disappointed!

There are several compelling reasons to run a .NET 8 web application in Docker:

Portability and Consistency: Docker creates standardized environments, ensuring your application runs the same way regardless of the underlying operating system. This simplifies deployment across different platforms, cloud providers, and development machines. It is easy to target both mainstream CPU architectures of our times (ARM and X86).

Isolation and Resource Management: Each container runs in a self-contained space, preventing conflicts with other applications and ensuring each receives the necessary resources. This improves reliability and security.

Simplified Development and DevOps: Docker streamlines the development workflow by providing a consistent environment for developers and testers. Additionally, containerized .NET applications integrate seamlessly with DevOps tools and CI/CD pipelines, automating deployment and management.

Creating a simple web app

I would like to use Linux for this because we are in the process of migrating from Windows to Linux at work and I wanted to have some exposure how hard is to set up the development environment.

• First step is to install the Micrososft repo that has the dotnet SDK. I use Debian for most Linux installations and the following works on Bookworm (x86). The special thing about my setup is the use of doas instead of sudo.

wget https://packages.microsoft.com/config/debian/12/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
doas apt install ./packages-microsoft-prod.deb
rm packages-microsoft-prod.deb
doas apt update
doas apt install dotnet-sdk-8.0 -y

Dotnet version can be verfied the following way:

l1x@deby ~> dotnet --version
8.0.200

Great, we have the dotnet version we need.

• Creating a web application

I opt out from the Microsoft data collection and try to figure out how to create a new project skeleton for a simple api.

l1x@deby ~> export DOTNET_CLI_TELEMETRY_OPTOUT=1
l1x@deby ~> dotnet new
The 'dotnet new' command creates a .NET project based on a template.

Common templates are:
Template Name   Short Name  Language    Tags
--------------  ----------  ----------  ----------------------
Blazor Web App  blazor      [C#]        Web/Blazor/WebAssembly
Class Library   classlib    [C#],F#,VB  Common/Library
Console App     console     [C#],F#,VB  Common/Console

An example would be:
   dotnet new console

Display template options with:
   dotnet new console -h
Display all installed templates with:
   dotnet new list
Display templates available on NuGet.org with:
   dotnet new search web

As it turns out creating a simple web api is really not that hard.

l1x@deby ~/c/dotnet-test> dotnet new webapi
The template "ASP.NET Core Web API" was created successfully.

Processing post-creation actions...
Restoring /home/l1x/code/dotnet-test/dotnet-test.csproj:
  Determining projects to restore...
  Restored /home/l1x/code/dotnet-test/dotnet-test.csproj (in 21.68 sec).
Restore succeeded.

The web application code is straightforward. The only thing I have added is the compression.

using System.IO.Compression;
using Microsoft.AspNetCore.ResponseCompression;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

builder.Services.AddResponseCompression(options => {
  options.EnableForHttps = true;
  options.Providers.Add < BrotliCompressionProvider > ();
  options.Providers.Add < GzipCompressionProvider > ();
});

var app = builder.Build();

// if (app.Environment.IsDevelopment()) {
app.UseSwagger();
app.UseSwaggerUI();
// }

app.UseResponseCompression();
app.UseHttpsRedirection();

var summaries = new [] {
  "Freezing",
  "Bracing",
  "Chilly",
  "Cool",
  "Mild",
  "Warm",
  "Balmy",
  "Hot",
  "Sweltering",
  "Scorching"
};

app.MapGet("/weatherforecast", () => {
    var forecast = Enumerable.Range(1, 5).Select(index =>
        new WeatherForecast(
          DateOnly.FromDateTime(DateTime.Now.AddDays(index)),
          Random.Shared.Next(-20, 55),
          summaries[Random.Shared.Next(summaries.Length)]
        ))
      .ToArray();
    return forecast;
  })
  .WithName("GetWeatherForecast")
  .WithOpenApi();

app.Run();

record WeatherForecast(DateOnly Date, int TemperatureC, string ? Summary) {
  public int TemperatureF => 32 + (int)(TemperatureC / 0.5556);
}

Creating Dockerfiles

I am assuming that docker is already installed on the system. If not the following packages should be installed:

containerd.io/bookworm,now 1.6.28-1 amd64 [installed,automatic]
docker-ce-cli/bookworm,now 5:25.0.3-1~debian.12~bookworm amd64 [installed]
docker-ce/bookworm,now 5:25.0.3-1~debian.12~bookworm amd64 [installed]

The first docker file is using the defaut images and stages for the build:

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build-env
WORKDIR /app
COPY . ./
RUN dotnet restore
RUN dotnet publish -c Release -o out

FROM mcr.microsoft.com/dotnet/aspnet:8.0
WORKDIR /app
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "dotnet-test.dll"]

Using this:

l1x@deby ~/c/dotnet-test (main)> docker build -t dotnet-test:default .
[+] Building 19.6s (14/14) FINISHED                                                                                                                                 docker:default
 => [internal] load build definition from Dockerfile                                                                                                                          0.2s
 => => transferring dockerfile: 300B                                                                                                                                          0.0s
 => [internal] load metadata for mcr.microsoft.com/dotnet/aspnet:8.0                                                                                                          0.0s
 => [internal] load metadata for mcr.microsoft.com/dotnet/sdk:8.0                                                                                                             0.0s
 => [internal] load .dockerignore                                                                                                                                             0.2s
 => => transferring context: 2B                                                                                                                                               0.0s
 => [build-env 1/5] FROM mcr.microsoft.com/dotnet/sdk:8.0                                                                                                                     1.4s
 => [stage-1 1/3] FROM mcr.microsoft.com/dotnet/aspnet:8.0                                                                                                                    0.8s
 => [internal] load build context                                                                                                                                             0.7s
 => => transferring context: 12.23MB                                                                                                                                          0.2s
 => [stage-1 2/3] WORKDIR /app                                                                                                                                                0.5s
 => [build-env 2/5] WORKDIR /app                                                                                                                                              0.2s
 => [build-env 3/5] COPY . ./                                                                                                                                                 0.2s
 => [build-env 4/5] RUN dotnet restore                                                                                                                                        8.3s
 => [build-env 5/5] RUN dotnet publish -c Release -o out                                                                                                                      8.1s
 => [stage-1 3/3] COPY --from=build-env /app/out .                                                                                                                            0.3s
 => exporting to image                                                                                                                                                        0.3s
 => => exporting layers                                                                                                                                                       0.3s
 => => writing image sha256:265648afb3c8fecba0a1c22ad196b552096819eeccf4444d0ec0f7adec678b31                                                                                  0.0s
 => => naming to docker.io/library/dotnet-test:default                                                                                                                        0.0s

We have the new buildx in docker but it seems the dangling images are still a thing. I guess these are required for caching.

l1x@deby ~/c/dotnet-test (main)> docker images
REPOSITORY                        TAG                  IMAGE ID       CREATED          SIZE
dotnet-test                       default              265648afb3c8   28 seconds ago   221MB
<none>                            <none>               cf116155bb30   2 minutes ago    221MB

Having a 221MB container is a not that bad.

Cleaning up the dangling images is easy:

docker rmi $(docker images -f "dangling=true" -q)

There are three versions of base images that I would like to use:

• Debian (this is the default)

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build-env
WORKDIR /app
COPY . ./
RUN dotnet restore
RUN dotnet publish -c Release -o out

FROM mcr.microsoft.com/dotnet/aspnet:8.0-bookworm-slim
WORKDIR /app
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "dotnet-test.dll"]

• Alpine 3.18

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build-env
WORKDIR /app
COPY . ./
RUN dotnet restore
RUN dotnet publish -c Release -o out

FROM mcr.microsoft.com/dotnet/aspnet:8.0-alpine3.18
WORKDIR /app
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "dotnet-test.dll"]

• Alpine 3.19

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build-env
WORKDIR /app
COPY . ./
RUN dotnet restore
RUN dotnet publish -c Release -o out

FROM mcr.microsoft.com/dotnet/aspnet:8.0-alpine3.19
WORKDIR /app
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "dotnet-test.dll"]

I would like to build all of these at one. For the CI/CD there are going to be separate steps.

cat build.all.sh
#!/usr/bin/env bash
docker build -t dotnet-test:net8.alpine3.18 -f Dockerfile.net8.alpine3.18 .
docker build -t dotnet-test:net8.alpine3.19 -f Dockerfile.net8.alpine3.19 .
docker build -t dotnet-test:net8.bookworm-slim -f Dockerfile.net8.bookworm-slim .

docker images
REPOSITORY                        TAG                  IMAGE ID       CREATED          SIZE
dotnet-test                       net8.alpine3.19      8730308faf31   54 seconds ago   111MB
dotnet-test                       net8.alpine3.18      694a028c16e6   55 seconds ago   110MB
dotnet-test                       default              265648afb3c8   9 minutes ago    221MB
dotnet-test                       net8.bookworm-slim   265648afb3c8   9 minutes ago    221MB
mcr.microsoft.com/dotnet/sdk      8.0                  54ed1faefb92   3 days ago       866MB
mcr.microsoft.com/dotnet/aspnet   8.0                  6eedb7553b12   6 days ago       217MB
mcr.microsoft.com/dotnet/aspnet   8.0-bookworm-slim    6eedb7553b12   6 days ago       217MB
mcr.microsoft.com/dotnet/aspnet   8.0-alpine3.18       20dce08103c1   6 days ago       106MB
mcr.microsoft.com/dotnet/aspnet   8.0-alpine3.19       baa6902e9e30   6 days ago       107MB

It is time for a little performance check.

Simple performance check

Using docker run with simple port mapping is the easiest to expose the service port and run the performance test. For this test I use a new Gravitron box (c7g.2xlarge, 8 CPU, 16G RAM) on AWS.

[ec2-user@ip-10-20-1-235 ~]$ docker run -p 8000:8000 -ti dotnet-test:net8.alpine3.19
warn: Microsoft.AspNetCore.Hosting.Diagnostics[15]
      Overriding HTTP_PORTS '8080' and HTTPS_PORTS ''. Binding to values defined by URLS instead 'http://0.0.0.0:8000'.
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://0.0.0.0:8000
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app

Results

• Alpine 3.19

[ec2-user@ip-10-20-1-235 ~]$ drill -q --benchmark benchmark.yml --stats
Concurrency 3000
Iterations 500000
Rampup 2
Base URL http://localhost:8000

Fetch weather             Total requests            500000
Fetch weather             Successful requests       500000
Fetch weather             Failed requests           0
Fetch weather             Median time per request   23ms
Fetch weather             Average time per request  25ms
Fetch weather             Sample standard deviation 22ms
Fetch weather             99.0'th percentile        115ms
Fetch weather             99.5'th percentile        134ms
Fetch weather             99.9'th percentile        424ms

Time taken for tests      11.2 seconds
Total requests            500000
Successful requests       500000
Failed requests           0
Requests per second       44550.11 [#/sec]
Median time per request   23ms
Average time per request  25ms
Sample standard deviation 22ms
99.0'th percentile        115ms
99.5'th percentile        134ms
99.9'th percentile        424ms

• Alpine 3.18

[ec2-user@ip-10-20-1-235 ~]$ drill -q --benchmark benchmark.yml --stats
Concurrency 3000
Iterations 500000
Rampup 2
Base URL http://localhost:8000

Fetch weather             Total requests            500000
Fetch weather             Successful requests       500000
Fetch weather             Failed requests           0
Fetch weather             Median time per request   24ms
Fetch weather             Average time per request  25ms
Fetch weather             Sample standard deviation 23ms
Fetch weather             99.0'th percentile        115ms
Fetch weather             99.5'th percentile        132ms
Fetch weather             99.9'th percentile        420ms

Time taken for tests      11.3 seconds
Total requests            500000
Successful requests       500000
Failed requests           0
Requests per second       44315.59 [#/sec]
Median time per request   24ms
Average time per request  25ms
Sample standard deviation 23ms
99.0'th percentile        115ms
99.5'th percentile        132ms
99.9'th percentile        420ms

• Bookworm

[ec2-user@ip-10-20-1-235 ~]$ drill -q --benchmark benchmark.yml --stats
Concurrency 3000
Iterations 500000
Rampup 2
Base URL http://localhost:8000


Fetch weather             Total requests            500000
Fetch weather             Successful requests       500000
Fetch weather             Failed requests           0
Fetch weather             Median time per request   23ms
Fetch weather             Average time per request  25ms
Fetch weather             Sample standard deviation 20ms
Fetch weather             99.0'th percentile        96ms
Fetch weather             99.5'th percentile        116ms
Fetch weather             99.9'th percentile        395ms

Time taken for tests      11.1 seconds
Total requests            500000
Successful requests       500000
Failed requests           0
Requests per second       45201.00 [#/sec]
Median time per request   23ms
Average time per request  25ms
Sample standard deviation 20ms
99.0'th percentile        96ms
99.5'th percentile        116ms
99.9'th percentile        395ms

The difference is probably due to Musl / Glibc.

What is Bun?

Bun is a toolkit for JavaScript and TypeScript applications, packaged as a single executable named "bun." It acts as a runtime, manages packages, bundles code, and runs tests, ensuring a smooth and effective development workflow. Developed in Zig and powered by JavaScriptCore, it drastically cuts down startup times and memory consumption.

Three things stand out to me:

1. All-in-One Solution: Bun provides a toolkit, including a runtime, package manager, bundler, and test runner. If you prefer having a cohesive and integrated development environment without the need to integrate multiple tools separately, Bun can be a convenient choice.

2. Performance Optimization: Bun is designed with a focus on performance. If you are working on projects where performance optimization is a critical concern, and you need a runtime and development environment that is tuned for speed, Bun might offer advantages over the default Node.js setup.

3. Simplified Development: Bun aims to simplify the development process, making it more straightforward and efficient. If you are looking for a tool that abstracts away complexities and provides a more streamlined workflow, Bun could be a suitable option.

Let's continue with the installation and setup for the AWS Lambda environment.

Installing Bun

For having the bun cli tool installed we can use Brew or something similar on different operating systems.

❯ brew install bun
==> Downloading https://formulae.brew.sh/api/formula.jws.json
################ 100.0%
==> Downloading https://formulae.brew.sh/api/cask.jws.json
################ 100.0%
==> Fetching oven-sh/bun/bun
==> Summary
🍺  /opt/homebrew/Cellar/bun/1.0.6: 7 files, 48.8MB, built in 2 seconds
==> Running `brew cleanup bun`...

Running it without any parameter:

❯ bun
Bun: a fast JavaScript runtime, package manager, bundler and test runner. (1.0.6)

  run       ./my-script.ts       Run JavaScript with Bun, a package.json script, or a bin
  test                           Run unit tests with Bun
  x         bun-repl             Install and execute a package bin (bunx)
  repl                           Start a REPL session with Bun

  init                           Start an empty Bun project from a blank template
  create    astro                Create a new project from a template (bun c)

  install                        Install dependencies for a package.json (bun i)
  add       zod                  Add a dependency to package.json (bun a)
  remove    redux                Remove a dependency from package.json (bun rm)
  update    tailwindcss          Update outdated dependencies
  link                           Link an npm package globally
  unlink                         Globally unlink an npm package
  pm                             More commands for managing packages

  build     ./a.ts ./b.jsx       Bundle TypeScript & JavaScript into a single file

  upgrade                        Get the latest version of Bun
  bun --help                     Show all supported flags and commands

  Learn more about Bun:          https://bun.sh/docs
  Join our Discord community:    https://bun.sh/discord

Getting Bun source code

This step is needed to build a Lambda layer.

❯ git clone git@github.com:oven-sh/bun.git
Cloning into 'bun'...
❯ cd bun/
❯ git checkout bun-v1.0.6
Note: switching to 'bun-v1.0.6'.
HEAD is now at 969da088f fix(install): re-evaluate overrides when removed
bun (969da08) via 🍞 v1.0.1 via ↯ v0.10.1

Installing lambda dependencies

There are a few dependencies that need to be installed before we can create a Lambda layer.

❯ cd packages/bun-lambda

❯ bun install
bun install v1.0.6 (969da088)
 + @oclif/plugin-plugins@3.9.1
 + bun-types@0.7.3
 + jszip@3.10.1
 + oclif@3.6.5
 + prettier@2.8.4
 + aws4fetch@1.0.17

 686 packages installed [6.68s]

❯ bun install @oclif/plugin-plugins
bun add v1.0.6 (969da088)

 installed @oclif/plugin-plugins@3.9.1

 4 packages installed [110.00ms]

Building and publishing the layer

With the dependencies installed we can create the Lambda layer for arm64.

❯ bun run build-layer -- \
        --arch aarch64   \
        --release latest \
        --output ./bun-lambda-layer.zip
$ bun scripts/build-layer.ts --arch aarch64 --release latest --output ./bun-lambda-layer.zip
Downloading... https://bun.sh/download/latest/linux/aarch64?avx2=true
Extracting...
Saving... ./bun-lambda-layer.zip
Saved

Publishing the layer is also very simple.

❯ bun run publish-layer --         \
  --layer bun-lambda-layer         \
  --arch aarch64                   \
  --release latest                 \
  --output ./bun-lambda-layer.zip  \
  --region eu-west-1

$ bun scripts/publish-layer.ts --layer bun-lambda-layer --arch aarch64 --release latest --output ./bun-lambda-layer.zip --region eu-west-1
Downloading... https://bun.sh/download/latest/linux/aarch64?avx2=true
Extracting...
Saving... ./bun-lambda-layer.zip
Saved
Publishing...
Published arn:aws:lambda:eu-west-1:651831719661:layer:bun-lambda-layer:1
Done

Checking the layer:

❯ aws lambda list-layers 
| jq '.Layers[].LayerArn' 
| xargs -I {} bash -c "aws lambda list-layer-versions --layer-name {} | jq '.LayerVersions[].LayerVersionArn'" 
| egrep bun
"arn:aws:lambda:eu-west-1:651831719661:layer:bun-lambda-layer:1"

Creating Lambda execution role

First we need a file with the assume role policy document.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

We can create the role called bun-lambda-role referencing the assume role policy.

❯ aws iam create-role --role-name bun-lambda-role --assume-role-policy-document file://role.policy.json
{
    "Role": {
        "Path": "/",
        "RoleName": "bun-lambda-role",
        "RoleId": "AROAZPRBSW3W7DL5YG6HZ",
        "Arn": "arn:aws:iam::651831719661:role/bun-lambda-role",
        "CreateDate": "2023-10-19T10:16:11+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Service": "lambda.amazonaws.com"
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    }
}

Creating the Lambda function

Creating the lambda function that handles the requests.

export default {
  async handler(request: Request): Promise<Response> {
    console.log(request.headers.get("x-amzn-function-arn"));
    return new Response(JSON.stringify({"Hello": "from Bun runtime"}), {
      status: 200,
      headers: {
        "Content-Type": "application/json",
      },
    });
  },
};

Saving this as index.ts and then creating a zip file.

zip bun-function.zip index.ts

Finally, we can create the function:

❯ aws lambda create-function \
          --function-name bun-lambda-function \
          --runtime provided.al2 \
          --architectures arm64 \
          --layers arn:aws:lambda:eu-west-1:651831719661:layer:bun-layer:1 \
          --zip-file fileb://bun-function.zip \
          --handler index.handler \
          --role arn:aws:iam::651831719661:role/bun-lambda-role

The response from the AWS API has all the details of the lambda function we just created:

{
    "FunctionName": "bun-lambda-function",
    "FunctionArn": "arn:aws:lambda:eu-west-1:651831719661:function:bun-lambda-function",
    "Runtime": "provided.al2",
    "Role": "arn:aws:iam::651831719661:role/bun-lambda-role",
    "Handler": "index.handler",
    "CodeSize": 365,
    "Description": "",
    "Timeout": 3,
    "MemorySize": 128,
    "LastModified": "2023-10-19T10:16:58.106+0000",
    "CodeSha256": "yNUNBaGkJuist/ZoQkZtnToNNXIU/taYQHqZrwCu3Mk=",
    "Version": "$LATEST",
    "TracingConfig": {
        "Mode": "PassThrough"
    },
    "RevisionId": "c8edb420-aeeb-4568-8289-ea7b0b845d00",
    "Layers": [
        {
            "Arn": "arn:aws:lambda:eu-west-1:651831719661:layer:bun-layer:1",
            "CodeSize": 33260946
        }
    ],
    "State": "Pending",
    "StateReason": "The function is being created.",
    "StateReasonCode": "Creating",
    "PackageType": "Zip",
    "Architectures": [
        "arm64"
    ],
    "EphemeralStorage": {
        "Size": 512
    },
    "SnapStart": {
        "ApplyOn": "None",
        "OptimizationStatus": "Off"
    },
    "RuntimeVersionConfig": {
        "RuntimeVersionArn": "arn:aws:lambda:eu-west-1::runtime:dce29199fb5887a2c4fceaa2f34d395ba43a74a6895b381cb9383b1c7f3b5875"
    }
}

To make the Lambda function accessible without any additional services we can enable function URL.

❯ aws lambda create-function-url-config \
  --function-name bun-lambda-function \
  --auth-type NONE
{
    "FunctionUrl": "https://wbjmhbkwx5ikumwxkhka4lcgle0nvjgs.lambda-url.eu-west-1.on.aws/",
    "FunctionArn": "arn:aws:lambda:eu-west-1:651831719661:function:bun-lambda-function",
    "AuthType": "NONE",
    "CreationTime": "2023-10-19T10:18:56.838382Z"
}

Testing the newly created function

curl -vIX GET https://wbjmhbkwx5ikumwxkhka4lcgle0nvjgs.lambda-url.eu-west-1.on.aws/
{"Message":"Forbidden"}⏎

It seems something is missing. Digging through the possible reasons I found the following:

Your function URL auth type is NONE, but is missing permissions required 
for public access. To allow unauthenticated requests, choose the Permissions 
tab and create a resource-based policy that grants lambda:invokeFunctionUrl 
permissions to all principals (*). Alternatively, you can update your function 
URL auth type to AWS_IAM to use IAM authentication.

Let's create the missing permission:

The JSON version of the policy:

{
  "Version": "2012-10-17",
  "Id": "default",
  "Statement": [
    {
      "Sid": "FunctionURLAllowPublicAccess",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "lambda:InvokeFunctionUrl",
      "Resource": "arn:aws:lambda:eu-west-1:651831719661:function:bun-lambda-function",
      "Condition": {
        "StringEquals": {
          "lambda:FunctionUrlAuthType": "NONE"
        }
      }
    }
  ]
}

Finally, we can invoke the function:

❯ curl -iX GET https://wbjmhbkwx5ikumwxkhka4lcgle0nvjgs.lambda-url.eu-west-1.on.aws/
HTTP/1.1 200 OK
Date: Thu, 19 Oct 2023 12:01:29 GMT
Content-Type: application/json
Content-Length: 28
Connection: keep-alive
x-amzn-RequestId: cd204d16-01c9-4468-a02e-9f6d7d3ee2e1
X-Amzn-Trace-Id: root=1-65311a98-3ead96f16ebd86bc6366d05b;sampled=0;lineage=fc7d7573:0

{"Hello":"from Bun runtime"}

Latency looks reasonable:

That’s all folks!

How fast DuckDB can query Parquet files?

I found mysqlf in the situation that I had to check few million lines of web logs in Parquet files and I have seen the raise of DuckDB in few articles already.

DuckDB is a high-performance, embedded analytical SQL database. It is designed to support efficient and fast querying of large datasets, including those stored in various file formats like Parquet. DuckDB can read Parquet files natively, without the need for conversion.

It is time to try it out!

Getting started

Installing DuckDB works as expected, there is an installer for most platforms:

brew install duckdb

Version:

❯ duckdb --version
v0.7.1 b00b93f0b1

Querying a bunch of files

The user interface is pretty similar to Sqlite3, because DuckDB builds on the top os Sqlite.

❯ duckdb
v0.7.1 b00b93f0b1
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D

The files are located in a folder. We have a file for each day.

❯ exa -ah *.parq
2023-03-01.gz.parq  2023-03-13.gz.parq
2023-03-02.gz.parq  2023-03-14.gz.parq
2023-03-03.gz.parq  2023-03-15.gz.parq
2023-03-04.gz.parq  2023-03-16.gz.parq
2023-03-05.gz.parq  2023-03-17.gz.parq
2023-03-06.gz.parq  2023-03-18.gz.parq
2023-03-07.gz.parq  2023-03-19.gz.parq
2023-03-08.gz.parq  2023-03-20.gz.parq
2023-03-09.gz.parq  2023-03-21.gz.parq
2023-03-10.gz.parq  2023-03-22.gz.parq
2023-03-11.gz.parq  2023-03-23.gz.parq
2023-03-12.gz.parq  2023-03-24.gz.parq

Lets just get the name and type of the fields:

D DESCRIBE SELECT * FROM read_parquet('*.gz.parq');
┌─────────────────────────────┬─────────────┬─────────┬─────────┬─────────┬─────────┐
│         column_name         │ column_type │  null   │   key   │ default │  extra  │
│           varchar           │   varchar   │ varchar │ varchar │ varchar │ varchar │
├─────────────────────────────┼─────────────┼─────────┼─────────┼─────────┼─────────┤
│ iso-datetime                │ VARCHAR     │ YES     │         │         │         │
│ x-edge-location             │ VARCHAR     │ YES     │         │         │         │
│ sc-bytes                    │ BIGINT      │ YES     │         │         │         │
│ c-ip                        │ VARCHAR     │ YES     │         │         │         │
│ cs-method                   │ VARCHAR     │ YES     │         │         │         │
│ cs-host                     │ VARCHAR     │ YES     │         │         │         │
│ cs-uri-stem                 │ VARCHAR     │ YES     │         │         │         │
│ sc-status                   │ VARCHAR     │ YES     │         │         │         │
│ cs-referer                  │ VARCHAR     │ YES     │         │         │         │
│ cs-user-agent               │ VARCHAR     │ YES     │         │         │         │
│ cs-uri-query                │ VARCHAR     │ YES     │         │         │         │
│ cs-cookie                   │ VARCHAR     │ YES     │         │         │         │
│ x-edge-result-type          │ VARCHAR     │ YES     │         │         │         │
│ x-edge-request-id           │ VARCHAR     │ YES     │         │         │         │
│ x-host-header               │ VARCHAR     │ YES     │         │         │         │
│ cs-protocol                 │ VARCHAR     │ YES     │         │         │         │
│ cs-bytes                    │ BIGINT      │ YES     │         │         │         │
│ time-taken                  │ DOUBLE      │ YES     │         │         │         │
│ x-forwarded-for             │ VARCHAR     │ YES     │         │         │         │
│ ssl-protocol                │ VARCHAR     │ YES     │         │         │         │
│ ssl-cipher                  │ VARCHAR     │ YES     │         │         │         │
│ x-edge-response-result-type │ VARCHAR     │ YES     │         │         │         │
│ cs-protocol-version         │ VARCHAR     │ YES     │         │         │         │
│ fle-status                  │ VARCHAR     │ YES     │         │         │         │
│ fle-encrypted-fields        │ VARCHAR     │ YES     │         │         │         │
│ c-port                      │ VARCHAR     │ YES     │         │         │         │
│ time-to-first-byte          │ DOUBLE      │ YES     │         │         │         │
│ x-edge-detailed-result-type │ VARCHAR     │ YES     │         │         │         │
│ sc-content-type             │ VARCHAR     │ YES     │         │         │         │
│ sc-content-len              │ BIGINT      │ YES     │         │         │         │
│ sc-range-start              │ VARCHAR     │ YES     │         │         │         │
│ sc-range-end                │ VARCHAR     │ YES     │         │         │         │
├─────────────────────────────┴─────────────┴─────────┴─────────┴─────────┴─────────┤
│ 32 rows                                                                 6 columns │
└───────────────────────────────────────────────────────────────────────────────────┘
D

I would like to generate reader statistics by the URLs that start with posts. DuckDB has amazing support for working with strings beyond the basic SQL features. One way of filtering results is to use a GLOB.

D SELECT "cs-uri-stem" FROM read_parquet('*.gz.parq')
WHERE "cs-uri-stem" GLOB '/posts/*' LIMIT 10;
┌──────────────────────────────────────────────────────────────────┐
│                           cs-uri-stem                            │
│                             varchar                              │
├──────────────────────────────────────────────────────────────────┤
│ /posts/2021/03/22/using-ldap-in-docker-with-caching/             │
│ /posts/2021/03/22/using-ldap-in-docker-with-caching              │
│ /posts/2020/05/08/why-i-chose-fsharp-for-our-aws-lambda-project/ │
│ /posts/2023/02/28/using-python-3.11-with-aws-lambda/             │
│ /posts/2023/02/28/using-python-3.11-with-aws-lambda              │
│ /posts/2023/02/28/using-python-3.11-with-aws-lambda/             │
│ /posts/2023/02/28/using-python-3.11-with-aws-lambda/             │
│ /posts/2023/02/28/using-python-3.11-with-aws-lambda/             │
│ /posts/2023/02/28/using-python-3.11-with-aws-lambda/             │
│ /posts/2023/02/28/using-python-3.11-with-aws-lambda/             │
├──────────────────────────────────────────────────────────────────┤
│                             10 rows                              │
└──────────────────────────────────────────────────────────────────┘

I would like to have the statistics so that the closing / does not matter and I am only interested in the last bit of the URL"

SELECT COUNT(1) AS cnt,
split_part(rtrim("cs-uri-stem", '/'), '/', 6) AS url
FROM read_parquet('*.gz.parq')
WHERE "cs-uri-stem" GLOB '/posts/*'
GROUP BY url
HAVING url <> ''
ORDER BY cnt DESC LIMIT 15;

┌────────┬──────────────────────────────────────────────────────────┐
│  cnt   │                           url                            │
│ int64  │                         varchar                          │
├────────┼──────────────────────────────────────────────────────────┤
│ 109023 │ using-llama-with-m1-mac                                  │
│   4372 │ using-python-3.11-with-aws-lambda                        │
│    711 │ misusing-ninja                                           │
│    287 │ beyond-the-borrow-checker                                │
│    142 │ getting-started-with-firecracker-on-raspberry-pi         │
│    135 │ new-tools                                                │
│     93 │ compressing-data-with-parquet                            │
│     90 │ running-asp.net-web-application-with-falco-on-aws-lambda │
│     89 │ using-ldap-in-docker-with-caching                        │
│     83 │ diving-into-firecracker-with-alpine                      │
│     80 │ compressing-aws-s3-logs-after-getting-hackernewsed       │
│     30 │ why-i-chose-fsharp-for-our-aws-lambda-project            │
│     25 │ how-long-will-the-worlds-uranium-supplies-last           │
│     23 │ matching-binary-patterns                                 │
│     18 │ freenas-11.3-upgrade-issues                              │
├────────┴──────────────────────────────────────────────────────────┤
│ 15 rows                                                 2 columns │
└───────────────────────────────────────────────────────────────────┘

I was wondering what DuckDB does under the hood:

EXPLAIN ...
┌─────────────────────────────┐
│┌───────────────────────────┐│
││       Physical Plan       ││
│└───────────────────────────┘│
└─────────────────────────────┘
┌───────────────────────────┐
│           TOP_N           │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│           Top 15          │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│       count(1) DESC       │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         PROJECTION        │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│            cnt            │
│             1             │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│       HASH_GROUP_BY       │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│             #0            │
│        count_star()       │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         PROJECTION        │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│            url            │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           FILTER          │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│((cs-uri-stem ~~~ '/posts/*│
│      ') AND (COALESCE     │
│(array_extract(string_...  │
│-uri-stem, '/'), '/'),...  │
│         ) != ''))         │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│           EC: 0           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│        PARQUET_SCAN       │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│        cs-uri-stem        │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│           EC: 0           │
└───────────────────────────┘

DuckDB is pretty great at reading Parquet files but how about writing?

D COPY (SELECT * FROM read_parquet('*.gz.parq'))
TO 'output.parquet' (FORMAT PARQUET, CODEC 'ZSTD');
100% ▕████████████████████████████████████████████████████████████▏

It does not dissapoint.

What about the performance? There is a built in way to analyze performance.

EXPLAIN ANALYZE
COPY (SELECT * FROM read_parquet('*.gz.parq'))
TO 'output.parquet' (FORMAT PARQUET, CODEC 'ZSTD');
┌─────────────────────────────────────┐
│┌───────────────────────────────────┐│
││         Total Time: 4.91s         ││
│└───────────────────────────────────┘│
└─────────────────────────────────────┘

D EXPLAIN ANALYZE
SELECT COUNT(1) AS cnt,
split_part(rtrim("cs-uri-stem", '/'), '/', 6) AS url
FROM read_parquet('*.gz.parq')
WHERE "cs-uri-stem" GLOB '/posts/*'
GROUP BY url
HAVING url <> ''
ORDER BY cnt
DESC LIMIT 15;

┌─────────────────────────────┐
│┌───────────────────────────┐│
│└───────────────────────────┘│
└─────────────────────────────┘
┌─────────────────────────────────────┐
│┌───────────────────────────────────┐│
││    Query Profiling Information    ││
│└───────────────────────────────────┘│
└─────────────────────────────────────┘
EXPLAIN ANALYZE
SELECT
    COUNT(1) AS cnt,
    split_part(rtrim("cs-uri-stem", '/'), '/', 6) AS url
    FROM read_parquet('*.gz.parq')
    WHERE "cs-uri-stem" GLOB '/posts/*'
    GROUP BY url
    HAVING url <> ''
    ORDER BY cnt DESC LIMIT 15;
┌─────────────────────────────────────┐
│┌───────────────────────────────────┐│
││        Total Time: 0.0727s        ││
│└───────────────────────────────────┘│
└─────────────────────────────────────┘
┌───────────────────────────┐
│      EXPLAIN_ANALYZE      │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│             0             │
│          (0.00s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           TOP_N           │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│           Top 15          │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│       count(1) DESC       │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│             15            │
│          (0.00s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         PROJECTION        │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│            cnt            │
│             1             │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│             46            │
│          (0.00s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│       HASH_GROUP_BY       │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│             #0            │
│        count_star()       │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│             46            │
│          (0.01s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         PROJECTION        │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│            url            │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│           115262          │
│          (0.07s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│           FILTER          │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│((cs-uri-stem ~~~ '/posts/*│
│      ') AND (COALESCE     │
│(array_extract(string_...  │
│-uri-stem, '/'), '/'),...  │
│         ) != ''))         │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│           EC: 0           │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│           115262          │
│          (0.11s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│        PARQUET_SCAN       │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│        cs-uri-stem        │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│           EC: 0           │
│   ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│           563607          │
│          (0.04s)          │
└───────────────────────────┘

This is insane performance for a M1 Mac for an average dataset.

It is a really nice way of working with a local tool without too much hassle. The performance that you can get is amazing.

The large language models wars

With the increasing interest in artificial intelligence and its use in everyday life, numerous exemplary models such as Meta's LLaMA, OpenAI's GPT-3, and Microsoft's Kosmos-1 are joining the group of large language models (LLMs). The only problem with such models is the you can't run these locally. Up until now. Thanks to Georgi Gerganov and his llama.cpp project it is possible to run Meta's LLaMA on a single computer without a dedicated GPU.

Running LLaMA

There are multiple steps involved in running LLaMA locally on a M1 Mac. I am not sure about other platforms or other OSes so in this article we are focusing only the aforementioned combination.

Step 1: Downloading the model

The official way is to request the model via this web form and download it afterward.

There is a PR open in the repository, that describes an alternative way (that is probably a violation of the terms of service).

https://github.com/facebookresearch/llama/pull/73

Anyways, after you downloaded the model (or more like models because there are a few different kinds of models in the folder) you should have something like this:

❯ exa --tree
.
├── 7B
│  ├── checklist.chk
│  ├── consolidated.00.pth
│  └── params.json
├── 13B
│  ├── checklist.chk
│  ├── consolidated.00.pth
│  ├── consolidated.01.pth
│  └── params.json
├── 30B
│  ├── checklist.chk
│  ├── consolidated.00.pth
│  ├── consolidated.01.pth
│  ├── consolidated.02.pth
│  ├── consolidated.03.pth
│  └── params.json
├── 65B
│  ├── checklist.chk
│  ├── consolidated.00.pth
│  ├── consolidated.01.pth
│  ├── consolidated.02.pth
│  ├── consolidated.03.pth
│  ├── consolidated.04.pth
│  ├── consolidated.05.pth
│  ├── consolidated.06.pth
│  ├── consolidated.07.pth
│  └── params.json
├── tokenizer.model
└── tokenizer_checklist.chk

As you can see the different models are in a different folders. Each model has a params.json that contains details about the model.

For example:

{
  "dim": 4096,
  "multiple_of": 256,
  "n_heads": 32,
  "n_layers": 32,
  "norm_eps": 1e-06,
  "vocab_size": -1
}

Step 2: Installing dependencies

Xcode must be installed to compile the C++ project. If you don't have it, please do the following:

xcode-select --install

These are the dependencies for building the C++ project (pkgconfigand cmake).

brew install pkgconfig cmake

Finally, we can install Torch.

I assume you have Python 3.11 installed so you can create a virtual env like this:

/opt/homebrew/bin/python3.11 -m venv venv

Activating the venv. I am using fish. For other shells just drop the .fish suffix.

. venv/bin/activate.fish

After activated the venv we can install Pytorch:

pip3 install --pre torch torchvision --extra-index-url https://download.pytorch.org/whl/nightly/cpu

If you are interesting leveraging the new Metal Performance Shaders (MPS) backend for GPU training acceleration you can verify it by running the following. This is not required for running LLaMA on you M1 though:

❯ python
Python 3.11.2 (main, Feb 16 2023, 02:55:59) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch; torch.backends.mps.is_available()
True

Now lets compile llama.cpp.

Step 3: Compile LLaMA CPP

Cloning the repo:

git clone git@github.com:ggerganov/llama.cpp.git

After installing all the dependencies you can run make:

❯ make
I llama.cpp build info:
I UNAME_S:  Darwin
I UNAME_P:  arm
I UNAME_M:  arm64
I CFLAGS:   -I.              -O3 -DNDEBUG -std=c11   -fPIC -pthread -DGGML_USE_ACCELERATE
I CXXFLAGS: -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -pthread
I LDFLAGS:   -framework Accelerate
I CC:       Apple clang version 14.0.0 (clang-1400.0.29.202)
I CXX:      Apple clang version 14.0.0 (clang-1400.0.29.202)

cc  -I.              -O3 -DNDEBUG -std=c11   -fPIC -pthread -DGGML_USE_ACCELERATE   -c ggml.c -o ggml.o
c++ -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -pthread -c utils.cpp -o utils.o
c++ -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -pthread main.cpp ggml.o utils.o -o main  -framework Accelerate
./main -h
usage: ./main [options]

options:
  -h, --help            show this help message and exit
  -s SEED, --seed SEED  RNG seed (default: -1)
  -t N, --threads N     number of threads to use during computation (default: 4)
  -p PROMPT, --prompt PROMPT
                        prompt to start generation with (default: random)
  -n N, --n_predict N   number of tokens to predict (default: 128)
  --top_k N             top-k sampling (default: 40)
  --top_p N             top-p sampling (default: 0.9)
  --temp N              temperature (default: 0.8)
  -b N, --batch_size N  batch size for prompt processing (default: 8)
  -m FNAME, --model FNAME
                        model path (default: models/llama-7B/ggml-model.bin)

c++ -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -pthread quantize.cpp ggml.o utils.o -o quantize  -framework Accelerate

Step 4: Converting the model

Assuming you placed the models under models/ in the llama.cpp repo.

python convert-pth-to-ggml.py models/7B 1

You should see an output like this:

{'dim': 4096, 'multiple_of': 256, 'n_heads': 32, 'n_layers': 32, 'norm_eps': 1e-06, 'vocab_size': 32000}
n_parts =  1
Processing part  0
Processing variable: tok_embeddings.weight with shape:  torch.Size([32000, 4096])  and type:  torch.float16
Processing variable: norm.weight with shape:  torch.Size([4096])  and type:  torch.float16
  Converting to float32
Processing variable: output.weight with shape:  torch.Size([32000, 4096])  and type:  torch.float16
Processing variable: layers.0.attention.wq.weight with shape:  torch.Size([4096, 4096])  and type:  torch.f
loat16
Processing variable: layers.0.attention.wk.weight with shape:  torch.Size([4096, 4096])  and type:  torch.f
loat16
Processing variable: layers.0.attention.wv.weight with shape:  torch.Size([4096, 4096])  and type:  torch.f
loat16
Processing variable: layers.0.attention.wo.weight with shape:  torch.Size([4096, 4096])  and type:  torch.f
loat16
Processing variable: layers.0.feed_forward.w1.weight with shape:  torch.Size([11008, 4096])  and type:  tor
ch.float16
Processing variable: layers.0.feed_forward.w2.weight with shape:  torch.Size([4096, 11008])  and type:  tor
ch.float16
Processing variable: layers.0.feed_forward.w3.weight with shape:  torch.Size([11008, 4096])  and type:  tor
ch.float16
Processing variable: layers.0.attention_norm.weight with shape:  torch.Size([4096])  and type:  torch.float
16
...
Done. Output file: models/7B/ggml-model-f16.bin, (part  0 )

The next step would be to perform the quantizing:

./quantize ./models/7B/ggml-model-f16.bin ./models/7B/ggml-model-q4_0.bin 2

Output:

llama_model_quantize: loading model from './models/7B/ggml-model-f16.bin'
llama_model_quantize: n_vocab = 32000
llama_model_quantize: n_ctx   = 512
llama_model_quantize: n_embd  = 4096
llama_model_quantize: n_mult  = 256
llama_model_quantize: n_head  = 32
llama_model_quantize: n_layer = 32
llama_model_quantize: f16     = 1
...
layers.31.attention_norm.weight - [ 4096,     1], type =    f32 size =    0.016 MB
layers.31.ffn_norm.weight - [ 4096,     1], type =    f32 size =    0.016 MB
llama_model_quantize: model size  = 25705.02 MB
llama_model_quantize: quant size  =  4017.27 MB
llama_model_quantize: hist: 0.000 0.022 0.019 0.033 0.053 0.078 0.104 0.125 0.134 0.125 0.104 0.078 0.053 0.033 0.019 0.022

main: quantize time = 29389.45 ms
main:    total time = 29389.45 ms

Step5: Running the model

❯ ./main -m ./models/7B/ggml-model-q4_0.bin \
        -t 8 \
        -n 128 \
        -p 'The first president of the USA was '

main: seed = 1678615879
llama_model_load: loading model from './models/7B/ggml-model-q4_0.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 512
llama_model_load: n_embd  = 4096
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 32
llama_model_load: n_layer = 32
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 11008
llama_model_load: n_parts = 1
llama_model_load: ggml ctx size = 4529.34 MB
llama_model_load: memory_size =   512.00 MB, n_mem = 16384
llama_model_load: loading model part 1/1 from './models/7B/ggml-model-q4_0.bin'
llama_model_load: .................................... done
llama_model_load: model size =  4017.27 MB / num tensors = 291

main: prompt: 'The first president of the USA was '
main: number of tokens in prompt = 9
     1 -> ''
  1576 -> 'The'
   937 -> ' first'
  6673 -> ' president'
   310 -> ' of'
   278 -> ' the'
  8278 -> ' USA'
   471 -> ' was'
 29871 -> ' '

sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000


The first president of the USA was 57 years old when he assumed office (George Washington). Nowadays, the US electorate expects the new president to be more young at heart. President Donald Trump was 70 years old when he was inaugurated. In contrast to his predecessors, he is physically fit, healthy and active. And his fitness has been a prominent theme of his presidency. During the presidential campaign, he famously said he
 would be the “most active president ever” — a statement Trump has not yet achieved, but one that fits his approach to the office. His tweets demonstrate his physical activity.

main: mem per token = 14434244 bytes
main:     load time =  1311.74 ms
main:   sample time =   278.96 ms
main:  predict time =  7375.89 ms / 54.23 ms per token
main:    total time =  9216.61 ms

Enjoy!

Bird view of a beautiful night at Meteora, misty and fog, finely detailed, extremely ornate, octane render, Unreal Engine, Cinematic, Color Grading, Photoshoot, Gamma, White Balance, Neon, Light, Dark, Light Mode, Dark Mode, High Contrast, 5D, Multiverse, 32k, Super - Resolution, Megapixel, ProPhoto RGB, VR, Lonely, Good, Massive, Big, Spotlight, Frontlight, Halfrear Lighting, Backlight, Rim Lights, Rim Lighting, Artificial Lighting, Natural Lighting, Incandescent, Moody Lighting, Cinematic Lighting, Studio Lighting, Soft Lighting, Hard Lighting, volumetric Light, Volumetric Lighting, Volumetric, Contre - Jour, Lighting, Split Lighting, Beautiful Lighting, Global Illumination, Lumen Global Illumination, Screen Space Global Illumination, Ray Tracing Global Illumination, Materiality, Ambient Occlusion, Scattering, Glowing, Shadows, Rough, Ray Tracing Reflections, Lumen Reflections, Screen Space Reflections, Diffraction Grading, Chromatic Aberration, Scan Lines, Ray Traced, Ambient Occlusion, Anti - Aliasing, FXAA, TXAA, RTX, SSAO, Shaders, OpenGL - Shaders, GLSL - Shaders, Post Processing, Post - Production, Cel Shading, Tone Mapping, CGI, VFX, SFX, insanely detailed and intricate, hypermaximalist, elegant, hyper realistic, super detailed, photography, highly detailed and realistic, photoreal, Ultrarealistic, cinematography, hyper detailed, absolute realism, cinematic scene cinematic lighting, hyper resolution, Perfectionism, angelical, elegant, concept art, ultra - detailed, hypermaximalist, Epic, Photorealism, Very high detail, shot on professional camera, cinematic lighting, soft lighting, intricate details, depth of field, photography, cinematic shot, incredibly detailed, 4K. 8K, 16K --ar 16:9 --quality 2 --uplight --v 4

A Python Problem

AWS Lambda has a Python offering that is limited to version 3.9. It is challenging to use that version due to many things, but primarily because package maintainers do not necessarily cater to users who use older versions. And there is more. When you work with Python, what you must realize is that you usually have C, C++, Rust, or libc dependencies not to mention CPU architectures. The most useful Python libraries are written in C, C++, Fortran, or Rust. So when you try to deploy your code to a less used platform, it can burst into flames in the worst kind of ways.

I have spent more hours trying to get some Python lib work on a platform than learning Rust.

The last nail in the coffin of trying to use Python 3.9 on AWS was when a library was using Rust that was compiled against a newer version of libc that AWS has. That triggered the creation of a new solution for deploying Python to AWS Lambda so that we do not need to care about these issues anymore.

Docker for the Rescue

By using a Docker image as the source for your Lambda, you can simplify your deployment. Docker allows you to package your application and its dependencies into a single container. In addition, you can deploy your entire application stack with a single command, making the deployment process simpler.

Docker provides a consistent environment for your application to run in, regardless of the underlying infrastructure. By using a Docker image as the source for your AWS Lambda function, you can ensure that your application will always run in the same environment, regardless of where it is deployed. Using a Docker image ensures that your application stack, including the operating system, libraries, and dependencies, is consistent across all environments. This consistency ensures that your application will behave the same way regardless of where it is deployed, which reduces the risk of errors or unexpected behavior.

How to Do It

1. Create, build, and upload a base image with the required Python version
2. Create, build, and upload an application-specific image
3. Set up and deploy the Lambda function with Terraform

Base Dockerfile

This Dockerfile constitutes our base image from which we build our application-specific image. This provides a stable environment for the app. To make our end image smaller, we start a multi-phase build. We specify the Python version we require for our runtime in the FROM lines. Then, we install the AWS Lambda Runtime Interface Client to guarantee communication between the Lambda environment and our code. In the second phase, we only need to copy the runtime interface client to the completed image.

ARG FUNCTION_DIR="/var/task"

FROM python:3.11.2-slim-bullseye as build-image
ARG FUNCTION_DIR

RUN mkdir -p ${FUNCTION_DIR} && mkdir -p /venv
RUN useradd -m -u 5000 lambda || :
RUN chown lambda ${FUNCTION_DIR} && chown lambda /venv
RUN apt-get update && \
  apt-get install -y --no-install-recommends \
  g++ \
  make \
  cmake \
  unzip \
  libcurl4-openssl-dev && \
  apt-get clean && \
  rm -rf /var/lib/apt/lists/*
USER lambda
RUN python -m venv /venv
ENV PATH="/venv/bin:$PATH"
RUN which pip
RUN pip install pip --upgrade
RUN pip install awslambdaric

FROM python:3.11.2-slim-bullseye
RUN mkdir -p /venv
RUN useradd -m -u 5000 lambda || :
RUN chown lambda /venv
USER lambda
COPY --from=build-image /venv /venv
ENV PATH="/venv/bin:$PATH"
RUN pip list
ENTRYPOINT [ "python", "-m", "awslambdaric" ]

As you can see we use a user instead of using root for running our app. Running your application as a non-root user is recommended. In the context of AWS Lambda security is probably not that big of a deal but following the principle of least privilege (PoLP) is a good idea.

We use Ninja for building and uploading images. You can find more about this here: Misusing Ninja. After being built, the base image is uploaded to AWS ECR, so it can be fetched for the application-specific image. This is our build.ninja file:

aws_account_id = xxxxxxxxxx
python_version = 3.11.2
repo = my-repo

rule login-to-ecr
  command = aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin ${aws_account_id}.dkr.ecr.eu-west-1.amazonaws.com
  description = Logging into ECR

rule build-aws-lambda-base
  command = docker build . -t ${repo}:lambda-py-${python_version} --file Dockerfile.aws_lambda_python_${python_version}
  description = Building ${repo}:lambda-py-${python_version}

rule tag-aws-lamda-base
  command = docker tag ${repo}:lambda-py-${python_version} ${aws_account_id}.dkr.ecr.eu-west-1.amazonaws.com/${repo}:lambda-py-${python_version}
  description = Tagging ${repo}:lambda-py-${python_version}

rule push-aws-lamda-base
  command = docker push ${aws_account_id}.dkr.ecr.eu-west-1.amazonaws.com/${repo}:lambda-py-${python_version}
  description = Push ${repo}:lambda-py-${python_version}

build login-to-ecr: login-to-ecr
build build-aws-lambda-base: build-aws-lambda-base || login-to-ecr
build tag-aws-lamda-base: tag-aws-lamda-base || build-aws-lambda-base
build push-aws-lamda-base: push-aws-lamda-base || tag-aws-lamda-base

default login-to-ecr build-aws-lambda-base tag-aws-lamda-base push-aws-lamda-base

Let's run it:

ninja -f build.ninja

We have the base image in ECR. This gives us control over what goes into the application image later. For immutable infrastructure it is paramount to use a base image that is not a moving target, meaning you do not run apt-get update or a similar command every time you build an application-specific container.

Application-Specific Dockerfile

This is the Dockerfile for the application that runs inside the Lambda function as a container. The base image, that has just been created, is referenced on the first line.

FROM <aws_account_id>.dkr.ecr.eu-west-1.amazonaws.com/<repo>:lambda-py-3.11.2
ARG FUNCTION_DIR="/var/task"
ENV PATH="/venv/bin:$PATH"
USER lambda
WORKDIR ${FUNCTION_DIR}
ADD app .
COPY pyproject.toml .
COPY README.md .
RUN pip install .
RUN pip list
CMD ["app.handler"]

In this file when we install the python packages we usually lock the versions to a specific one, again, we would like to have immutable infra, running the build on different computers should result in the same image.

Our directory looks like the following:

|__app/
| |__app.py
| |__pyproject.toml
| |__README.MD
|
|__Dockerfile
|__build.ninja

If you have a more complicated application there are many more files in the app folder.

Our entry point is the function called handler that is inside the app.py. CMD specifies where is the Lambda handler and will be used by the ENTRYPOINT of the base image.

This image is built similarly to the base image: with Ninja. This is our build.ninja file for the app-specific image:

version   = 0.6.1
aws_account_id = xxxxxxxx
repo = my-repo


rule login-to-ecr
  command = aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin ${aws_account_id}.dkr.ecr.eu-west-1.amazonaws.com
  description = Logging in to ECR

rule build-image
  command = docker build . -t ${repo}:backend-api-${version} --file Dockerfile
  description = Building ${repo}:backend-api-${version}

rule tag-image
  command = docker tag ${repo}:backend-api-${version} ${aws_account_id}.dkr.ecr.eu-west-1.amazonaws.com/${repo}:backend-api-${version}
  description = Tagging ${repo}:backend-api-${version}

rule upload-image
  command = docker push ${aws_account_id}.dkr.ecr.eu-west-1.amazonaws.com/${repo}:backend-api-${version}
  description = Push ${repo}:backend-api-${version}


build login-to-ecr: login-to-ecr
build build-image: build-image || login-to-ecr
build tag-image: tag-image || build-image
build upload-image: upload-image || tag-image

default login-to-ecr build-image tag-image upload-image

Let's run it:

ninja -f build.ninja

Now there is an image in ECR that hosts our application. You can very easily move back and forth between different base image versions, try out new libraries, or update your application. Because we deploy the same image to dev and prod you can gain confidence about the version that you are working on as it passes through different stages from local to dev and finally to prod.

I have left out testing and formatting, linting from the build process because it would be too much to display the complete build file with those steps. We use pytest, Ruff, and Black for most of these tasks. We also implemented integration tests in pytest for more thorough testing.

Lambda Function with Terraform

Our final task is to deploy to AWS Lambda. We are using Terraform to achieve this. The package_type must be Image and the image URI needs to be provided. We take adventage of ARM64 with this setup which is a bit cheaper than running on AMD64.

resource "aws_lambda_function" "docker-lambda-function" {
  function_name = "my-lambda-function"
  description   = "This is my lambda function"
  image_uri     = "<account_id>.dkr.ecr.eu-west-1.amazonaws.com/<repo>:backend-api-<lambda-function-version>"
  package_type  = "Image"
  role          = <lambda_role_arn>
  memory_size   = 2048
  timeout       = 10
  architectures = ["arm64"]
}

After deploying to AWS Lambda we also create a CloudFront distribution and API Gateway v2 api that fronts Lambda. The new function-url makes it possible to skip API Gateway. Maybe later on we can have a look what are the tradeoffs with that setup.


## Conclusion

Using a Docker image as the packaging for an AWS Lambda function provides a consistent, immutable environment for your application, ensuring that it runs the same way regardless of where it is deployed. This can help reduce errors and unexpected behavior and make it easier to package and distribute your application and its dependencies. Python 3.11 is much faster in many cases than previous versions of Python.

Abstract

In this blog post, we explore the reasons why Rust is an excellent choice for Extract, Transform, Load (ETL) jobs. I have been writing ETL jobs for small and big companies in Java, Scala, and Python for over a decade. Python is an obvious winner of this segment and its popularity shows up in StackOverflow surveys. Because of its popularity there is unlimited content on the internet on how to do certain tasks in Python and you can get help on the usual platforms very easily. However, Python has some pretty big downsides.

First of all, being a dynamic interpreted language makes it pretty hard to trust your code. The chance that it works for you and fails in production is non-trivial. Docker and other methods help you with this but why would you need to use another tool just to have reliable deployments? I have wasted a lot of hours on Python deployment issues and it takes a lot to have a reliable setup that smaller teams usually don't have.

This is where Rust comes into the picture.

Rust's unique features like static cross-platform building, a single way to configure projects, simple dependency management, memory safety, and built-in support for serialization and deserialization make it an attractive option for data processing tasks. The borrow checker ensures that memory is managed correctly, making it almost impossible to write code with memory errors, resulting in reliable and safe ETL jobs.

We usually run ETL jobs on execution platforms like k8s with Airflow or AWS Glue with Spark. These platforms are usually pretty slow and inefficient for smaller tasks and do not yield great performance out of the box. Companies end up hiring experts who can fine-tune these systems to specific workloads.

If you're looking for a modern and efficient alternative for your ETL jobs, Rust is worth considering.

Why on Earth would anybody use Rust for ETL?

• Static cross-platform building

Rust's cross-platform building feature allows developers to create binary executables that can run on different operating systems. This is because Rust produces a single binary that can be deployed across multiple platforms. This feature makes it easy to write ETL jobs that can be run on different systems.

• Single way to configure your project

Rust has a great approach to configuring projects. Instead of relying on a variety of configuration files, Rust uses a single configuration file, Cargo.toml, to configure the project. You can, of course, try to add more configuration files if you want but the official way is Cargo.toml. Python has at least three different ways. This makes it easy to manage and maintain the project's dependencies and configurations.

• Handling dependency versions are simple

One of the most significant challenges in developing software is managing dependencies. Rust solves this problem by using the Cargo package manager. Cargo allows developers to declare dependencies and their versions in the Cargo.toml file. Cargo then downloads and installs the required dependencies, ensuring that they work together correctly. While Python has many issues with different operating system dependencies and Python versions, Rust does not. I have seen a problem once where a dependency had some issues but it was trivial to fix and the workaround was obvious. I cannot say the same about Python where I run into problems with libraries not supporting a certain combination of Python and operating system.

• It is almost impossible to write broken libraries

Rust has a unique feature that ensures that developers write the correct code. The borrow checker ensures that memory is managed correctly, making it almost impossible to write code with memory errors. On top of that the strict type system makes it trivial to check type safety across the code and this makes it easy to write reliable and safe ETL jobs.

• Serialization and deserialization are simple

Rust has built-in support for the serialization and deserialization of data. This feature makes it easy to read and write data in various formats, making it an excellent choice for ETL jobs.

• Async makes parallel and concurrent workflows easy to write

Python also has some async capabilities but I like Rust a bit more. I think the people who designed Rust async did a great job from a practical point of view. I know it could be better and I am aware of its largely theoretical criticism.

Ok, so let us have a look at a side-by-side comparison of writing a medium size ETL job in both languages.

Types

First, let's create a type in each language to represent the problem at hand. We would like to create a lookup structure with two ways lookup and store strings and hashes in it.

• Python

Just to have something easy to work with you need to use a library in Python. It has class-based higher-level types which are a bit harder to use in practice than the Rust equivalent.

from pydantic import BaseModel
class EidCache(BaseModel):
    forward_dict: Dict[str, str]
    reverse_dict: Dict[str, str]
    created_at: Optional[str]

• Rust

The Rust version is roughly the same.

#[derive(Serialize, Deserialize, Debug)]
pub struct EidCache {
    pub forward_dict: BTreeMap<String, String>,
    pub reverse_dict: BTreeMap<String, String>,
    created_at: String,
}

Code

This is my first Rust code, so go easy on me. :) I am not sure how to reduce the memory allocation, maybe use references instead of creating a new string every time.

• Python

Python is a bit easier to write if you are already familiar with dictionary comprehension.

def get_reverse_dictionary(d: Dict[str, str]):
    return {v: k for k, v in d.items()}


def create_db_eid_cache():
    database_names = get_all_database_names()
    forward_dict = {name: get_db_eid(name) for name in database_names}
    reverse_dict = get_reverse_dictionary(forward_dict)

    return EidCache(
        forward_dict=forward_dict, reverse_dict=reverse_dict, created_at=iso_utc_now()
    )

• Rust

fn create_eid_cache<F>(names: Option<Vec<String>>, eid_fun: F) -> Option<EidCache>
where
    F: Fn(&str) -> String,
{
    match names {
        Some(xs) => {
            let xs_with_eid: Vec<(String, String)> =
                xs.iter().map(|x| (x.to_string(), eid_fun(x))).collect();

            let mut forward_dict: BTreeMap<String, String> = BTreeMap::new();
            let mut reverse_dict: BTreeMap<String, String> = BTreeMap::new();

            xs_with_eid.iter().for_each(|x| {
                forward_dict.insert(x.0.clone(), x.1.clone());
                reverse_dict.insert(x.1.clone(), x.0.clone());
                ()
            });

            let created_at = OffsetDateTime::now_utc().format(&Iso8601::DEFAULT).unwrap();

            let request_count = 0;

            Some(EidCache {
                forward_dict,
                reverse_dict,
                created_at,
                request_count,
            })
        }
        None => {
            let msg = "There are no name entries".to_string();
            info!("{}", msg);
            None
        }
    }
}

As you can see the codes are comparable in length, the Rust version being a bit longer.

The whole project is a 1150 lines of Rust and 687 lines of Python. It involves talking to different AWS APIs, calculating checksums and write summary data to S3. Because it is easy to use multiple files in Rust we split up the code into smaller more managable chunks and compile it to a single binary before deploying the function to Lambda. With Python we need to zip up the files and go through all sorts of hoops because AWS supports only Python 3.7 or 3.10 depending on which Glue version is available. The dependency management is much easier with Rust, we simple need to use the Cargo.toml for it while in the Python case you have to provide your dependencies which get installed at run time adding more time to the already slow execution.

Performance

Our Python code runs around 2 minutes and 30 seconds (p50) because it has to start up the Spark cluster and do lots of things that we do not need. I was trying to understand how would it look like if we move this workload out of Glue into Lambda and used Rust instead of Python. I could have used Python on Lambda as well and I am going to write an article about that experience next. Anyways, our first attempt with Rust was extremely smooth thanks to cargo-lambda and the language tooling. The p50 value for the AWS Lambda function with Rust is 1,3 seconds. Yes this is an apples to oranges comparison.

I think that is a pretty big improvement that we cannot ignore. As far as developer productivity goes: yes you can write Python faster than Rust but you are going to pay the penalty later at deployment time and when you run in production.

Summary

I am pretty happy with Rust and we are going to use it more and more. I am ok to have a bit more development time and significantly reduce the operation efforts we have to put in to maintain the system.

Abstract

In the world of software development, build systems are essential tools for compiling and linking source code into executables. While there are many build systems available (make, cmake), one of the most popular and powerful is the Ninja build system.

Developed by Evan Martin while working on the Chromium project at Google, Ninja is a fast, scalable, and cross-platform build system that has gained widespread adoption in the software development community. In this blog post, we'll take a closer look at the Ninja build system and its key features.

How Ninja Works

At its core, Ninja is a simple, low-level build system that aims to be fast and efficient. Ninja works by generating a graph of dependencies between input files and output files and then executing a series of build commands to transform the input files into the output files.

The input files are typically source code files written in a programming language such as C++, Rust, or in our case Python. The output files are the compiled object files, libraries, and executables that result from the build process. And this is where it gets interesting. You can safely ignore this part of Ninja and just use it as a dag orchestrator that executes commands in a certain order.

Ninja uses a file format called "build.ninja" to define the build graph and specify the build commands. The build.ninja file is a text file that describes the dependencies between the input and output files or in our case the build steps.

Key Features of Ninja

One of the main advantages of Ninja is its speed. Because Ninja generates a build graph that only includes the necessary build steps, it can execute builds quickly and efficiently. Additionally, Ninja can scale to handle large projects with many dependencies, making it a popular choice for building complex software applications. For our use case, it is not that much use but in case you are using it for a C++ project it is pretty good.

Another key feature of Ninja is its cross-platform support. Ninja is designed to work on multiple operating systems, including Windows, macOS, and Linux, which makes it easy to use in a variety of development environments. This is true if you have a way of making sure that the tools that are used in the build are present and have the same CLI on all systems you run your builds on.

Getting Started with Ninja

I usually learn by example and there aren't many examples around. I have read the original documentation and tried a few things out but finally, I understand how all things hang together.

There are 3 things that you need to know about a ninja build file:

• rule
• build
• dependency

A rule is a description and a command that gets executed when the rule is invoked.

A build step is just calling a rule (or potentially multiple rules using dependencies).

A dependency is a concept of describing a relationship between rules.

Let's have a look at a basic example.

rule ok
  command = echo "ok"

build ok: ok

Invoking it:

❯ ninja ok
[1/1] echo "ok"
ok

Let's add a build that has two steps (sometimes called stages or targets).

rule one
  command = echo "one"

rule two
  command = echo "two"

build one: one
build two: two || one

With two pipe characters, we can express dependency between the build stages.

Invoking the build:

❯ ninja one
[1/1] echo "one"
one

❯ ninja two
[1/2] echo "one"
one
[2/2] echo "two"
two

This means we cannot forget to execute the first step when executing the second.

Using Ninja for building and deploying to the cloud

There is no point in writing an article without Docker in it so let's put Docker into Ninja.

version   = 0.6.0
aws_account_id = 11111111111


rule login-to-ecr
  command = aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin ${aws_account_id}.dkr.ecr.eu-west-1.amazonaws.com
  description = Logging in to ECR

rule init
  command = env | egrep AWS_PROFILE
  description = Displaying AWS_PROFILE


rule build-image
  command = docker build . -t depoxy:backend-api-${version} --file Dockerfile
  description = Building depoxy:backend-api-${version}

rule tag-image
  command = docker tag depoxy:backend-api-${version} ${aws_account_id}.dkr.ecr.eu-west-1.amazonaws.com/depoxy:backend-api-${version}
  description = Tagging depoxy:backend-api-${version}

rule upload-image
  command = docker push ${aws_account_id}.dkr.ecr.eu-west-1.amazonaws.com/depoxy:backend-api-${version}
  description = Push depoxy:backend-api-${version}



build login-to-ecr: login-to-ecr
build init: init || login-to-ecr
build build-image: build-image || init
build tag-image: tag-image || build-image
build upload-image: upload-image || tag-image

default login-to-ecr init build-image tag-image upload-image

There goes, a simple docker workflow implemented in Ninja. We use similar workflows to upload files to AWS and deploy with Terraform too. This unified our different build efforts mostly using YAML for CI/CD and made sure we do not forget anything while not needing to write a tonne of YAML.

The reduction from the YAML-based workflows is pretty significant:

+229 −1,633

To get started with Ninja

To get started with Ninja, you'll need to install the Ninja build system on your development machine. Ninja can be installed from package managers on the most popular operating systems, or you can download the source code and build it manually.

Once you have Ninja installed, you'll need to create a build.ninja file that describes your project's build process. You can create this file manually, or you can use a build system generator such as gn or Meson to generate it automatically.

Finally, you can run the Ninja build command to execute the build process and create the output files. Ninja will automatically track changes to input files and re-execute the build commands as needed, ensuring that your output files are always up-to-date.

Conclusion

The Ninja build system is a powerful and efficient tool for building software applications. With its speed, scalability, and cross-platform support, Ninja is a great choice for developers working on a wide range of projects. Whether you're building a small utility or a large, complex application, Ninja can help you streamline your build process and get your software up and running quickly. It helped us to move from YAML to a much better alternative while reducing the complexity of the CI/CD workflows. The next is to find a CI/CD platform that supports Ninja files or maybe to create a project that does exactly that.

I am amazed by the amount of new software that is high quality, good-looking, and easy to use. Here are some of the favorites that I have encountered. Some of these (probably the majority) are written in Rust.

List of new tools

Starship

Starship is a simple, fast, and customizable cross-platform prompt for command line shells. It is designed to be lightweight and efficient, while still offering a wide range of customization options to suit your individual needs. With Starship, you can easily switch between different shells and operating systems without having to worry about how to carry your prompt with you. Whether you are a beginner or an experienced command line user, Starship is an excellent tool to improve your productivity and make your workflow more seamless.

You might be familiar with the following already, but this is what it looks like:

link

Alacritty

Alacritty is a high-performance terminal emulator that offers a range of customization options while still maintaining sensible defaults.

These are some of Alacritty's noteworthy features:

• Vi Mode — Move around and create selections using vi bindings
• Search — Search for any text within the scrollback buffer
• Regex Hints — Mark any text for mouse or keyboard interaction
• Multi-Window — Improve resource usage by using only a single Alacritty process

link

Helix

Helix is a new text editor that is designed to be fast, intuitive, and customizable. It features a modern, minimalist user interface. My favorite feature is the integration with Tree-sitter which is probably the fastest parser generator out there.

link

Ruff

After trying to use many other linters before, Ruff does a great job while being extremely fast.

Ruff aims to be orders of magnitude faster than alternative tools while
integrating more functionality behind a single, common interface.

link

Ripgrep

ripgrep is a tool that allows you to search for a specific pattern (expressed using a regular expression) within the files in a directory and its subdirectories. It is designed to be fast and efficient, and it has some helpful default behaviors, such as ignoring files and directories that are listed in a .gitignore file and skipping over hidden files and binary files. This makes it a convenient and reliable option for quickly searching through large codebases or other collections of files. It is usually much faster than grep or similar alternatives.

link

Exa

exa is a new version of the classic ls command that is included with Unix and Linux operating systems. It provides several additional features and improvements over the original ls, such as the use of color coding to distinguish different file types and metadata. If you are a command line user, you might find exa to be a helpful and modern replacement for the standard ls command.

link

Zola

Zola is a static content management system (CMS) written in Rust. Zola is designed to be fast, easy to use, and flexible, making it a good choice for building static websites and blogs. Some of the key features of Zola include its Markdown support using CommonMark, a strongly defined, highly compatible specification of Markdown, Sass compilation, syntax highlighting, and the ability to generate a table of contents. It is less featureful than Hugo though.

link

Duf

Duf is a disk usage utility (similar to the widely used df) for displaying free disk space in a visually more appealing manner.

Closing

These are the new tools that I use nowadays hopefully it is useful for some of you as well.

LDAP and Active Directory are both directory services. These are used extensively in corporate environments as authentication and authorization solutions. In a modern infrastructure, applications quite often run in a containerized environment, and sometimes these environments need to have access to LDAP or AD. In this article, we investigate how a Docker container can efficiently access these directory services.

Creating a base container

Ubuntu 18.04.5 LTS (Bionic Beaver) is a long-term release, and this is what we are going to use going forward. There must be a local user other than root for tasks like compiling code or installing NPM packages if necessary.

FROM ubuntu:bionic-20210222

RUN apt-get clean && apt-get update

RUN apt-get install --reinstall ca-certificates -y

RUN apt-get install sudo apt-transport-https ca-certificates -y

RUN useradd -m -u 5000 app || :
RUN echo 'app ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers

RUN apt-get clean

USER app

Creating a new tag is simple:

sudo docker build . -t local/aloha-base:2021.03.23 --file docker/0/Dockerfile

Sending build context to Docker daemon  229.9kB
Step 1/9 : FROM ubuntu:bionic-20210222
 ---> 329ed837d508
Step 2/9 : RUN apt-get clean && apt-get update
 ---> Using cache
 ---> 3fd8e7b8ccdc
Step 3/9 : RUN apt-get install --reinstall ca-certificates -y
 ---> Using cache
 ---> d1d2a538daf2
Step 4/9 : RUN apt-get install sudo apt-transport-https ca-certificates -y
 ---> Using cache
 ---> bc8142001e3e
Step 5/9 : RUN useradd -m -u 5000 admin || :
 ---> Running in 9939670ed4aa
Removing intermediate container 9939670ed4aa
 ---> ca9a0f390db3
Step 6/9 : RUN useradd -m -u 5001 app || :
 ---> Running in 68cd99b566b5
Removing intermediate container 68cd99b566b5
 ---> 37c5b5146df9
Step 7/9 : RUN echo 'admin ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
 ---> Running in a82a060ba035
Removing intermediate container a82a060ba035
 ---> d3e527406739
Step 8/9 : RUN apt-get clean
 ---> Running in 9e4d7b7ac071
Removing intermediate container 9e4d7b7ac071
 ---> 117422e5341e
Step 9/9 : USER app
 ---> Running in b28f471b4c5d
Removing intermediate container b28f471b4c5d
 ---> 5f025336ed65
Successfully built 5f025336ed65
Successfully tagged local/aloha-base:2021.03.23

This image has the two users that we need. One is an app that will be used to run the application process and admin that can be used to execute further modifications (installing packages, changing config, etc.) if necessary. I usually split Docker container building into multiple separate Dockerfiles, because quite often, these steps take long enough to have a real impact on the overall build time. In some cases, I cannot rely on Docker caching, so I have a few steps, and the last step is application-specific, reducing the CI/CD time to the absolute minimum.

Creating the S6 container (based on base)

S6 is a lightweight process supervision suite for managing long-lived processes in a UNIX-like operating system or in a container. The question if multi-process containers are a good idea or not is not in scope here.

FROM local/aloha-base:2021.03.23

USER admin

RUN sudo apt update
RUN sudo DEBIAN_FRONTEND=noninteractive apt install libpam-ldap unscd software-properties-common -y

RUN sudo add-apt-repository ppa:deadsnakes/ppa
RUN sudo apt update
RUN sudo apt install python3.9 -y

ADD https://github.com/just-containers/s6-overlay/releases/download/v2.2.0.3/s6-overlay-amd64-installer /tmp/
RUN sudo chmod +x /tmp/s6-overlay-amd64-installer && sudo /tmp/s6-overlay-amd64-installer /