# FluxPipe: Serverless Flux / InfluxDB

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1665829066748/XkAZJNV9E.png align="left")

# Flux
[Flux](https://docs.influxdata.com/flux/v0.x/) is an open source functional data scripting language from [InfluxDB](https://docs.influxdata.com) designed for _querying, analyzing, and acting on data_. **Flux** supports multiple data source types, including Time series databases _(such as InfluxDB)_, Relational SQL databases _(such as MySQL, Postgres or ClickHouse)_, HTTP APIs and CSV datasources.

**Flux** unifies _querying, processing, writing, and acting on data_ into a single syntax. The flux language is designed to be _usable, readable, flexible, composable, testable, contributable, and shareable._ You can even extend flux using flux scripts!

> Flux is part of InfluxDB 1.7 and 2.0, but **can be run independently of those**.

Unfortunately, *no stand-alone version* of Flux existed in the wild .... _until now!_


### <a href="https://metrico.in"><img src="https://user-images.githubusercontent.com/1423657/162720189-976cc0cc-7511-4278-a942-9c4e7cc9148a.png" height="200" style="height: 200px;"></a>

### Hello, FluxPipe
[FluxPipe](https://github.com/metrico/fluXpipe) is an experimental stand-alone **Flux API** built for serverless and used by our tools at [qxip/metrico](https://metrico.in). Made in _rust/golang_ it offers a thin API layer _(emulating a basic InfluxDB 2.x endpoint)_ and near instant startup and query processing times.

Just like we did for [ClickHouse](https://blog.qryn.dev/wrapping-a-serverless-clickhouse-pt1) the Fluxpipe binary is built to portable, static and ready to run anywhere - _without storage_ - instantly compatible with most Flux [scripts](https://github.com/metrico/fluXpipe/tree/main/scripts) and ready to use and modify for an endless variety of data piping and processing purposes.

> What can we do with our fresh [portable Flux binary](https://fluxpipe.com)?

Of course you can run it on your servers large or small. _But that's so boring!_

<br>

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1665570893115/z576IZ16g.png align="left")

# Flux into Space

Today, we're launching [Flux](https://docs.influxdata.com/flux/v0.x/) into Space with [Deta](https://alpha.deta.space/ ) 🚀

<br>

> **Warnings**<br>
>_This is a free experimental ride for demo purposes. Not a final product!_<br>
>_The goal is to showcase the flexibility of Deta and power of Influx/Flux_<br>

<br>

## Sign up for [Deta Space Alpha](https://alpha.deta.space/)
Deta is your own personal computer in the cloud: _private, secure & always online_

🟣 All apps on Deta Space run in your own sandboxed 'personal cloud'<br>
🟣 App developers cannot see or touch your personal infrastructure<br>
🟣 App users run clones and remain in the drivers seat with all the data<br>

Deta Space Micros, Base and Drive are perfect for modern web applications.

_But can they run something as complex as Flux?_ &nbsp; **TLDR; HELL YEAH!**

<br>

## Launch your Flux Space
Ready? Browse to the [Flux Space](https://alpha.deta.space/discovery/flux-5qq-v0.187.1) page using Discovery and Install

<a href="https://alpha.deta.space/discovery/flux-5qq-0.0.2" target=_blank>
<img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1665735048581/O4-yP0oP-.png"></a>

<img src="https://d24bnpykhxwj9p.cloudfront.net/s3file/s3fs-public/users1/2017-02/Thu/stanley1.gif">


Space will generate a unique secure URL for your **Flux Space** API endpoint
```
https://FLUX-SPACE-URL.deta.app
```

<iframe src="https://fluxpipe.fly.dev" width=100% height=500></iframe>
<center>☝️☝️☝️ This is a Live demo, not just an image ☝️☝️☝️</center>


Awesome! Your serverless, stateless and _bottomless_ Flux API is ready! 🔥

##### **TLDR;** _test before reading using the preset selector_

<br>

### 📙 Flux Language

**FluxPipe** is designed to serve and execute *Flux scripts* for querying, analyzing, and acting on data. Flux [scripts](https://github.com/metrico/fluXpipe/tree/main/scripts) can range from extremely simple, to extremely complex, importing [standard and custom functions](https://docs.influxdata.com/flux/v0.x/stdlib/) available to extend the scope of execution.

### Let's try it out!

You can query your Flux endpoint using **CURL** or any Flux client such as [Grafana](https://grafana.com/docs/grafana/latest/datasources/influxdb/influxdb-flux/)

#### CURL
`POST` your fist query with curl to generate some _sample data._
```
curl -XPOST https://FLUX-SPACE-URL.deta.app/api/v2/query -sS \
  -H 'Accept:application/csv' \
  -H 'Content-type:application/vnd.flux' \
  -d 'import g "generate" g.from(start: 2022-04-01T00:00:00Z, stop: 2022-04-01T00:03:00Z, count: 3, fn: (n) => n)'
```

Congratulations! You're now a _Flux Space Cadet!_ 🏆🚀


#### Grafana
There's more! You can configure your **Flux Space** endpoint as a new `InfluxDB` datasource

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1665830832721/i9Pn7tXxm.png align="left")

**Save** and **Explore** data using _your own serverless datasource!_

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1665789401630/H1Z90PyQ3.png align="left")

<br>

> The world is your data oyster! _Let's start exploring some simple scrips._

#### Query MySQL
```
import "sql" 

sql.from(driverName: "mysql",dataSourceName: "default:password@tcp(127.0.0.1:9004)/system", query: "SELECT database, total_rows FROM tables") 
|> rename(columns: {database: "_value", total_rows: "_data"}) |> keep(columns: ["_value","_data"])
```

#### Query APIs
```
import "experimental/http/requests"
import ejson "experimental/json"
import "json"
import "array"

response =
     requests.post(
         url: "https://goolnk.com/api/v1/shorten",
         body: json.encode(v: {url: "http://metrico.in"}),
         headers: ["Content-Type": "application/json"],
     )

data = ejson.parse(data: response.body)
array.from(rows: [data])
```

#### Scrape Metrics
```
import "experimental/prometheus"
prometheus.scrape(url: "http://prometheus-exporter:9096/metrics")
```

... and there's [so much more you can do](https://github.com/metrico/fluXpipe/tree/main/scripts) using the excellent [flux documentation](https://docs.influxdata.com/flux/v0.x/)

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1665831474217/edAvx4wdD.png align="left")

<br>

> Now, you might know we're ClickHouse freaks around here so...


### Flux 💛 ClickHouse
Let's try a simple **ClickHouse** query via `HTTP API` + `CSVWithNames` format
```
import "csv"
import "experimental/http/requests"
import "strings"

url = "https://clickhouse.server"
query = "SELECT name, value FROM table FORMAT CSVWithNames"
response = requests.get(url:url, params: ["query": [query]])

csv.from(csv: string(v: response.body), mode: "raw")
```

That's clean and simple. Works anywhere.

Fluxpipe offers an _experimental_  ClickHouse binary `sql` driver  _(PR open)_

```
import "sql" 

sql.from(
  driverName: "clickhouse",
  dataSourceName: "clickhouse://default:@clickhouse-host:9000/system",
  query: "SELECT database, total_rows FROM tables WHERE total_rows > 0"
) 
|> rename(columns: {database: "_value", total_rows: "_data"})
|> keep(columns: ["_value","_data"])
```

### Sharing Data

Let's use [S3rver-Deta](https://alpha.deta.space/discovery/s3rver-1.0.6) Drive as shared CSV storage between **Flux** and **ClickHouse**

Insert some data in an **S3** Table using **ClickHouse**

```sql
INSERT INTO FUNCTION s3('S3RVER-SPACE-ENDPOINT.deta.app/test/flux.csv', 'CSV', 'name String, value UInt32') FORMAT Values ('one', 1), ('two', 2),('three', 3), ('four', 4), ('five',5);

Query id: ad61bcbc-a92b-4e55-8f71-2063d5c9488c

Ok.

5 rows in set. Elapsed: 0.495 sec. 
```

Let's read our data back from **S3** and parse it using **Flux**

```
import "csv"
import "experimental"
import "experimental/http/requests"
response = requests.get(
    url: "https://S3RVER-SPACE-ENDPOINT.deta.app/test/flux.csv",
)
csv.from(csv: string(v: response.body), mode: "raw")
```

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1665839492616/7HQOFDjKQ.png align="left")

> Et Voila' - works like magic!

<br>

_Pinch me awake! Is this is all powered by serverless, on-demand functions?!_ 🤯


#### Technical Notes

⚠️ Fluxpipe does not support internal `buckets` - only external datasources<br>
⚠️ Fluxpipe does not support `secrets` - use `ENV` variables instead<br>
⚠️ Certain scripts might fail - please [open an issue](https://github.com/metrico/fluXpipe/issues) and we'll take a look<br>


## Conclusion

This Flux server _does not exist_. It's a _serverless Grafana datasource from Space!_<br>

The question is: _What will you build with it?_ 🚀 🚀 🚀 

<a href="https://alpha.deta.space"><img src="https://res.cloudinary.com/crunchbase-production/image/upload/c_lpad,h_256,w_256,f_auto,q_auto:eco,dpr_1/hyxpdvyfvlvsz9dh3a70"></a>

Thanks to the **Deta Team** for their dedication and for letting us play with their alpha!

#### ▶ UPDATE 

* **Fluxpipe** can be deployed on [fly.io] and any other service supporting Docker
* **Fluxpipe** now has a Playground UI so you don't need Grafana to _try this out!_

<iframe src="https://fluxpipe.fly.dev" width=100% height=500></iframe>
<center>☝️☝️☝️ This is a Live demo, not just an image ☝️☝️☝️</center>

#### Want More?

_Did this get you excited? Join our team, [we're hiring](mailto:info@qxip.net)!_<br>
_Working with ClickHouse? Try our [polyglot observability stack](https://qryn.dev)_

<a href="https://metrico.in"><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1665785535618/6MMcraiG8.png"></a>

> Enjoy and Share your Comments! 👋