Posts
Processing CloudFront Logs with Rust
Turning gzipped AWS CloudFront access logs into Parquet with Rust and Polars: finding files with a glob, parsing the log lines and building a DataFrame.
Running .NET 8 web applications in Docker
Building a .NET 8 web API on Debian and running it in Docker: Debian and Alpine images compared (221 MB vs 111 MB), plus a quick load test.
Running Bun as AWS Lambda function
Running Bun on AWS Lambda: building and publishing the Bun Lambda layer, creating the execution role and the function, and testing the deployed handler.
How fast DuckDB can query Parquet files?
Querying daily Parquet files of web logs with DuckDB on an M1 Mac: a top-pages aggregation in 0.07 s and a full ZSTD Parquet rewrite in 4.9 s.
Using LLaMA with M1 Mac
Running Meta's LLaMA 7B on an M1 Mac using only the CPU: compiling llama.cpp, converting and quantizing the model with Python 3.11, and generating text.
Using Python 3.11 with AWS Lambda
AWS Lambda's Python runtime stopped at 3.9, so this runs Python 3.11 from a container image: a base image, an app image and deployment with Terraform.
Beyond the Borrow Checker
Why Rust suits ETL jobs: static builds, Cargo dependency management, serde and async, with a side-by-side Python and Rust version of the same job.
Misusing Ninja
Using the Ninja build system beyond compiling code: rules, builds and dependencies explained by example, then used to drive Docker builds and cloud deploys.
New tools
Command line tools worth switching to, most of them written in Rust: Starship, Alacritty, Helix, Ruff, ripgrep, exa, Zola and duf, and what each does well.
Using LDAP in Docker with caching
LDAP logins inside Ubuntu Docker containers with PAM and NSS, services supervised by S6, and unscd caching lookups to spare the directory server.
Compressing data with Parquet
Converting a 3 GB SQLite dataset of GitHub code snippets to Parquet: 427 MB with Snappy and 265 MB with Gzip, under 9% of the original size.
Compressing AWS S3 logs after getting HackerNewsed
Merging thousands of small S3 access logs into Parquet with an F# tool after a Hacker News spike, then charting referers and top posts with Plotly Dash.
Running ASP.Net web application with Falco on AWS Lambda
Running an F# web app built on Falco (ASP.NET Core) locally and on AWS Lambda, so the edit-compile-run loop no longer needs a deploy.
Diving into Firecracker with Alpine
Part 2 of the Firecracker series: installing Alpine Linux on a Raspberry Pi 4 and running Firecracker microVMs on a lightweight Alpine host.
Getting started with Firecracker on Raspberry Pi
Part 1 of the Firecracker series: setting up Ubuntu on a Raspberry Pi 4 and installing Firecracker, Jailer and Firectl to run KVM-based microVMs.
Why I chose Fsharp for our AWS Lambda project
Why F# won over Python and Rust for a small AWS Lambda that tracks website usage without third-party JavaScript, with the handler written in F#.
How long will the world’s uranium supplies last?
A back-of-the-envelope estimate of how long uranium could power the world, from the energy in 1 kg of U-235 to military stockpiles and US energy use.
FreeNAS 11.3 upgrade issues
The FreeNAS 11.3-U2.1 upgrade removed the applications in my jails. How to restore them by rolling the jail back to its pre-update snapshot with iocage.
Matching binary patterns
How Erlang bit syntax builds and matches binaries, with a complete example that crafts an NTP request and parses the response by pattern matching.
Small Alpine Linux containers with Java 13
Small Java Docker images with Project Portola: using jlink and the musl build of OpenJDK 13 on Alpine Linux to get a 41 MB Java runtime image.
Creating partitions automatically in PostgreSQL
Splitting a large PostgreSQL table into daily partitions with a PL/pgSQL trigger that creates each partition on first insert, using table inheritance.
Converting Amazon S3 logs to Avro
Reading Amazon S3 access logs with Clojure and the AWS Java SDK, and converting the text records to schema-backed Avro files for analytical queries.
High Performance Kafka Producer on AWS
How the new Apache Kafka producer works in sync and async modes, how it assigns records to partitions, and how to configure it for throughput on AWS.
Simple Kafka Consumer In Clojure
Building a Kafka consumer in Clojure with Shovel, a small wrapper around the Kafka client API: connector config, message streams and core.async threads.
Using custom schema with Riak Search 2.0
Indexing DBpedia city data in Riak Search 2.0 with a custom Solr schema: configuring the cluster, creating the index and bucket type, and querying Solr.