# chDB AWS Lambda Fun(ction)

Let's run [chDB](https://blog.qryn.dev/wrapping-a-serverless-clickhouse-pt3) as an **AWS Lambda** function using Docker and Python!

*If you're not yet familiar with* [*chDB*](https://chdb.io)*, here's a quick recap for you:*

[![](https://github.com/chdb-io/chdb/raw/pybind/docs/_static/snake-chdb.png align="left")](https://chdb.io)

> chDB is an embedded SQL OLAP Engine powered by ClickHouse

### Features

* In-process SQL OLAP Engine, powered by ClickHouse
    
* Serverless. No need to install ClickHouse
    
* Minimized data copy from C++ to Python with [python memoryview](https://docs.python.org/3/c-api/memoryview.html)
    
* Input&Output support Parquet, CSV, JSON, Arrow, ORC and 60+[more](https://clickhouse.com/docs/en/interfaces/formats) formats
    
* Support Python DB API 2.0, [example](https://github.com/chdb-io/.github/blob/main/examples/dbapi.py)
    
* Library bindings for [Python](https://github.com/chdb-io/chdb), [Go](https://github.com/chdb-io/chdb-go), [Rust](https://github.com/chdb-io/chdb-rust), [NodeJS](https://github.com/chdb-io/chdb-node), [Bun](https://github.com/chdb-io/chdb-bun)
    

The **chdb** library can run in-process and makes it super easy to run **ClickHouse SQL** queries in a variety of languages and can be used to mimic ClickHouse query APIs.

### TLDR; Here's a [Live Demo](https://fluxpipe.fly.dev)

<iframe src="https://chdb.fly.dev/?user=default#U0VMRUNUICdjaGRiJywgdmVyc2lvbigp" width="100%" height="500"></iframe>

<center>☝️☝️☝️ This is not an image. Click RUN for results ☝️☝️☝️</center>

## ⚡ Let's OLAP on Lambdas!

*Let's assemble our chDB Lambda. This article describes how to:*

1. Create and test an **AWS Lambda** function running chDB in Docker.
    
2. Use **chDB** to query any supported data source.
    
3. Use **SQL** to filter, process and downsample cloud data.
    
4. Write the data back to **ClickHouse, S3** or any supported destination.
    
5. Use **CloudWatch** or **EventBridge** to trigger your functions automatically.
    

## Create an AWS Lambda Function for chDB

To create an AWS Lambda function, log into your AWS console.

➡ **Build & Push** the latest [chDB-lambda container](https://github.com/chdb-io/chdb-server/tree/main/lambda) image to your **ECR** storage.

➡ Search for **AWS Lambda** and select the service. Then, click **Create Function**.

![create function](https://images.ctfassets.net/o7xu9whrs0u9/5XE0x5uBoOA4oJYZFwzNea/79493c8495c8d60c726cfeeae73a2b84/create_function.png align="left")

➡ Choose **Container Image** and use the **chDB ECR** instance URI you created

![](https://user-images.githubusercontent.com/1423657/250210923-887894c3-35ef-4083-a4b8-29d247f1fc1c.png align="left")

➡ Click **Create Function** at the bottom right when you’re done.

### Validation

Let's **test** our new **chDB Lambda** using a *simple query.*

The Lamba expects JSON requests with a **query** key:

```plaintext
{
   'query': 'SELECT version()',
   'default_format': 'CSV'
}
```

And the response would look like this (or any other format)

```plaintext
22.12.1.1
```

You can also use the Browser and the AWS Console to generate **test events**:

![image](https://user-images.githubusercontent.com/1423657/250201531-daa26b0b-68e2-4cec-b665-5505efe99b99.png align="left")

### 👏 Well Done!

*Once your Lambda is validated, the sky's the limit! Move your existing ClickHouse workflows into your lambdas and use them to downsample, forecast and report.*

* Connect to any dataset on S3/R2/Minio using Parquet, Arrow, etc
    
* Connect to any ClickHouse Server securely to pull/insert data
    
* Use your favorite programming language to work with ClickHouse
    
* Save money by running ClickHouse queries with Lambda workers
    

💡 For additional security, use [AWS Secrets Manager](https://aws.amazon.com/blogs/compute/securely-retrieving-secrets-with-aws-lambda/) and **ENV** variables to control sensible fields *(such as authentication ids, tokens, etc)* in your *ClickHouse scripts.*

### Automation

If you’re looking to perform a downsampling task, you’ll need to run your Lambda script on a schedule. You can use **CloudWatch** or **EventBridge** to create a rule and target your AWS Lambda function to run chDB scripts on a user-defined schedule.

Use the following documentation depending on your preferred service:

* [CloudWatch](https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html)
    
* [EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-run-lambda-schedule.html)
    

### **Conclusion**

Just a few clicks and you are all set with a *low-cost, high-power* **chDB Lambda** function ready to perform simple and complex data processing tasks at any scale.

[Use with Logs, Metrics and Traces for end-to-end visibility using qryn's polyglot API and data from Loki, Prometheus, Opentelemetry, InfluxDB, Elastic and many more.](https://qryn.cloud)

[![](https://cdn.hashnode.com/res/hashnode/image/upload/v1676413342775/491ef8fb-6764-4786-8d87-02d3a0e60642.png?auto=compress,format&format=webp&auto=compress,format&format=webp align="left")](https://qryn.cloud/)