Sven Eliasson

Kubernetes Trouble-Cheat-Sheet with 8 most important best practices (PDF, direct download)

Sven Eliasson — Sat, 18 May 2024 17:24:58 GMT

Whenever trouble arises, you better be prepared. To avoid wasting time trying to remember commands and potential fixes, I keep a text note with the most important information. These notes have served me well over the years, and I would like to share them in a printable format, also for myself.

Right after an incident is the second-best time to fix bad practices and prevent future issues. Learning from these moments can significantly improve your workflow and minimize future disruptions. To help with this, Ive compiled a list of eight best practices that I see violated pretty often and whose syntax I need to look up regularly.

I will update this article with more explanations in the future. For now, you can download the printable PDF :
Link: Download PDF (Github)

]]>

Benchmarking Hetzner's Storage Classes for Database Workloads on Kubernetes

Sven Eliasson — Sat, 27 Apr 2024 16:37:49 GMT

TLDR: Running Kubernetes on Hetzner offers cost-effective options, but handling production workloads, especially stateful ones like databases, raises concerns. Hetzner provides instance and cloud volume storage options with significant differences in IOPS performance. Longhorn, a distributed block storage system, can be used to leverage local volumes, but benchmarks show a slowdown compared to raw local files. Probably host a datatbase either on a dedicated host or use a hosted option instead.

Running Kubernetes on Hetzner is likely one of the most cost-effective options available. The Hetzeners Terrraform Kubernetes project simplifies the process of setting up a basic cluster within minutes. However, is this sufficient for handling production workloads?

One controversial topic about running a Kubernetes cluster is whether one should run stateful workloads like databases on Kubernetes. There are valid arguments for both sides. I used to avoid running databases on Kubernetes, but changed my mind. I think the widespread adoption of Kubernetes operators, including those for most databases, has changed the game. There are battle-proven operators that handle most of the pain points. However, there is one critical issue those operators cannot solve: IOPS.

You can increase the computing power of a database as much as you like, but if the IOPS are low, the database performance will suffer. Ultimately, you have to retrieve data from a disk, which significantly hampers query performance. In this short article, I intend to outline my benchmarks for different storage options.

Kubernetes on Hetzner storage options

There are two main types of storage options available on the Hetzner Cloud:

Instance volume storage: This is the NVMe storage linked to the node you rent. This volume scales only with the number of CPU and Memory you choose, making it quite costly.
Cloud volume storage: This is a block storage volume that is relatively inexpensive but offers lower performance.

The difference in IOPS is significant. For a medium-sized instance, the read IOPS on a local volume are 17 times higher than on cloud storage. You can find a detailed benchmark of IOPS performance for all instance types here: https://pcr.cloud-mercato.com/providers/hetzner/flavors/ccx21/performance/iops

In Hetzner's Kubernetes solution, there are three built-in storage classes that can be used out of the box:

hcloud-volumes: This is a block storage drive, the slower option.
longhorn-volumes: Longhorn is an open-source, distributed block storage system. It can optionally utilize the local drive of each node.
local-file: This is Rancher's implementation that simplifies the use of local host files as a volume.

Using local files in Kubernetes has several drawbacks and should generally be avoided. These files are inherently tied to specific hosts, complicating adaptation to changes in node architecture. This approach also introduces security vulnerabilities and significantly increases maintenance complexity.

Longhorn can be enabled during cluster installation using Terraform, and it aims to make use of local volumes. To achieve this, you must configure Longhorn as described here: https://gist.github.com/ifeulner/d311b2868f6c00e649f33a72166c2e5b

In summary: You need to enable the default disk on nodes first via settings.

defaultSettings:  createDefaultDiskLabeledNodes: true  kubernetesClusterAutoscalerEnabled: true # if autoscaler is active in the cluster  defaultDataPath: /var/lib/longhorn  # ensure pod is moved to an healthy node if current node is down:  node-down-pod-deletion-policy: delete-both-statefulset-and-deployment-pod

Label the nodes and assign a portion of the drive to a disk type "nvme":

kubectl label node <node> node.longhorn.io/create-default-disk='config'kubectl annotate node <storagenode> node.longhorn.io/default-disks-config='[ { "path":"/var/lib/longhorn","allowScheduling":true, "storageReserved":21474836240, "tags":[ "nvme" ]}, { "name":"hcloud-volume", "path":"/var/longhorn","allowScheduling":true, "storageReserved":10737418120,"tags":[ "ssd" ] }]'

Now you can define a longhorn storage clas0 that will utilize the nvme disk like this:

kind: StorageClassapiVersion: storage.k8s.io/v1metadata:  name: longhorn-fastprovisioner: driver.longhorn.ioallowVolumeExpansion: truereclaimPolicy: DeletevolumeBindingMode: Immediateparameters:  numberOfReplicas: "3"  staleReplicaTimeout: "2880" # 48 hours in minutes  fromBackup: ""  fsType: "ext4"  diskSelector: "nvme"

Ok, we are ready to go. Lets benchmark!

Benchmark storage IOPS for a given storage class in Kubernetes.

We need a simple solution to benchmark a given storage class. I used dbbench's docker container to spin up a simple benchmark job in kubernetes. First we need to define some sample PVCs to work with.

kind: PersistentVolumeClaimapiVersion: v1metadata:  name: dbench-pv-claim-localpathspec:  storageClassName: local-path  accessModes:    - ReadWriteOnce  resources:    requests:      storage: 10Gi---kind: PersistentVolumeClaimapiVersion: v1metadata:  name: dbench-pv-claim-longhornspec:  storageClassName: longhorn-fast  accessModes:    - ReadWriteOnce  resources:    requests:      storage: 25Gi--- kind: PersistentVolumeClaimapiVersion: v1metadata:  name: dbench-pv-claim-hcloudspec:  storageClassName: hcloud-volumes  accessModes:    - ReadWriteOnce  resources:    requests:      storage: 25Gi

We utilize kubernetes jobs to run the individual benchmark.

---apiVersion: batch/v1kind: Jobmetadata:  name: db-bench-01spec:  template:    spec:      containers:      - name: dbench        image: storageos/dbench:latest        env:          - name: DBENCH_MOUNTPOINT            value: /data        volumeMounts:        - name: dbench-pv          mountPath: /data      restartPolicy: Never      volumes:      - name: dbench-pv        persistentVolumeClaim:          claimName: dbench-pv-claim-longhorn           #claimName: dbench-pv-claim-hcloud           #claimName: dbench-pv-claim-localpath

A typical benchmark result will look like this:

Random Read/Write IOPS: 32.7k/27.1k. BW: 2854MiB/s / 2140MiB/sAverage Latency (usec) Read/Write: 325.47/216.90Sequential Read/Write: 5741MiB/s / 4689MiB/sMixed Random Read/Write IOPS: 23.8k/7946

I ran benchmarks for all storage options and got a rather disappointing result. Longhorn adds some overhead, causing a 4.5x slowdown compared to a raw local file, and only slightly outperforms a cloud volume.

I'm not sure if there are optimizations available for Longhorn. I halted my evaluation here; it appears that using a local file on the host node might be the best choice for a database workload.

In my upcoming post, I will share my experience of running a database workload on Hetzner in production using local files.

]]>

2.5x Performance: PostgreSQL to Clickhouse without Kafka with MaterializedPostgreSQL

Sven Eliasson — Thu, 25 Apr 2024 14:43:49 GMT

TLDR: Stream PostgreSQL data to ClickHouse without Kafka using MaterializedPostgreSQL for real-time data replication and optimization, achieving significant data compression and query 2.5x performance improvements.

Dealing with billions of sensor records, as we do at Rysta, is no easy task. We encountered challenges in maintaining the performance of some of our analytical workloads within PostgreSQL. PostgreSQL has been exceptional for us, and I suggest that anyone handling analytical tasks (OLAP) should start with PostgreSQL, specifically TimescaleDB.

However, for numerous analytical queries, PostgreSQL's row-based storage is slower compared to columnar storage used in many OLAP databases. After exploring various analytical database choices, I fell in love with ClickHouse. It excites me daily, and I started creating prototypes using ClickHouse, which delivered exceptional performance.

Currently, we store all our sensor records in a PostgreSQL database, and we have no plans to change this setup soon. Hence, we aim to replicate data from PostgreSQL to ClickHouse in real-time to power certain analytical workloads. A common method involves using CDC with Debezium/Kafka, as explained in this blog post. While this is a reliable and proven method, we do not utilize a Kafka cluster right now, and I prefer to avoid the extra work that comes with it. So, why not stream data directly into ClickHouse without an intermediary data broker? This is precisely achievable with ClickHouse's experimental feature called MaterializedPostgreSQL.

MaterializedPostgreSQL allows you to set up a ClickHouse database from a distant PostgreSQL database. Once an initial data snapshot is taken, modifications are copied using PostgreSQL's WAL. As of now, this feature is still experimental, but it has worked perfectly for us until this point.

SET allow_experimental_database_materialized_postgresql=1;CREATE DATABASE postgresENGINE = MaterializedPostgreSQL('host:port', 'database', 'user', 'password')SETTINGS materialized_postgresql_schema = 'myschema';

We used this approach to sync a sample dataset of about a billion records in about 5 minutes, which amazed me! The initial prototypes used a batch-wise ETL process that took over an hour for the first data sync. So already a good start! The table is automatically updated as new data is inserted into PostgreSQL.

Now that we have the date in our clickhouse cluster we could start using it... but it would probably not be faster than using a postgreSQL replica.

select sum(bytes_on_disk) FROM system.parts where database = 'postgres'/ 21841081469

The sample dataset uses around 20GB of data for a billion records. Let's utilize ClickHouse's specific features to optimize this dataset for quicker query performance:

Use codecs to compress the physical data size; smaller datasets are often faster to query.
Utilize a sparse index to reduce the number of chunks that need to be scanned line by line.
Implement a partition that will divide the table based on a pattern optimized for the queries we intend to execute.

There are trade-offs to consider, and the table structure should be fine-tuned based on your most crucial queries. The table DDL that proved to be most efficient was:

CREATE TABLE mydb.records (value Float32 CODEC(Gorilla),t DateTime CODEC(DoubleDelta),sensor_id UInt32 CODEC(T64)) ENGINE = ReplicatedMergeTreePARTITION BY toYYYYMM(t)  ORDER BY (toStartOfHour(t), sensor_value_id, t)PRIMARY KEY ( toStartOfHour(t), sensor_value_id);

Now that we have the perfect table, how do we populate it with data? I could insert data from the Postgres database into my ClickHouse-native table using an insert query, but this would have to be done for every insertion into the Postgres table. This is where we can make use of ClickHouse's "Materialized view," which functions differently compared to other databases like PostgreSQL. It's more like a trigger that runs a query on a specific table every time data is inserted into the source table.

CREATE MATERIALIZED VIEW mydb.records_mv TO mydb.records ASSELECT     data as value,     timestamp as t,     sensor_id as sensor_idFROM postgres.records;

After running this query, whenever new data is added to the PostgreSQL table, it will also be inserted into our ClickHouse-native, optimized table. This process only adds new records. However, we can initially copy over a snapshot like this:

INSERT INTO mydb.sensor_value_recordSELECT    data as value,     timestamp as t,     sensor_id as sensor_idFROM postgres.records

Voila! The resulting table compressed the initial 20GB into 3.9GB, achieving an 87% compression.

So that's it? Basically yes.
So everything is great? Not exactly...

CREATE MATERIALIZED VIEW mydb.records_mv_2024_04 TO mydb.records AS SELECT data as value, timestamp as t, sensor_id as sensor_idFROM postgres.records_2024_04;CREATE MATERIALIZED VIEW mydb.records_mv_2024_05 TO mydb.records AS SELECT data as value, timestamp as t, sensor_id as sensor_idFROM postgres.records_2024_05;CREATE MATERIALIZED VIEW mydb.records_mv_2024_06 TO mydb.records AS SELECT data as value, timestamp as t, sensor_id as sensor_idFROM postgres.records_2024_06;... and so on

I hope there will be native support for partitions in MaterializedPostgreSQL. However, from my research, it seems that this support is not actively being worked on or planned at the moment.

So, this is how we currently execute a prototype for particular analytical tasks, and it has been successful. In a specific slow query (retrieving all sensor data of a user in a given month), we noticed a 2.5x performance improvement!

If you have any questions, feel free to email me at sven.eliasson-at-gmail.com

]]>

Bun+ElysiaJS vs Node+Express.js Using a Single Real-World Request

Sven Eliasson — Sun, 17 Dec 2023 19:35:07 GMT

This is a brief overview of a basic performance test. I don't claim that my tests can be extrapolated to anything beyond the scope of this specific test.

Bun is the latest addition to the JavaScript runtime scene, challenging the dominance of Node.js in backend applications. Similar to another competitor, Deno, it compiles TypeScript by default. However, Bun promises to be significantly faster than Node.js and to serve as a direct, 1:1 replacement for Node.js. Back in September, when Bun reached version 1.0, I conducted a preliminary test with one of our APIs. The results showed similar performance to our existing Node.js (Express.js) app but with a 100% increase in memory usage. This wasn't a great start, but I hadn't implemented any of Bun's optimizations, and there were some bugs causing memory leaks. I decided to wait a bit longer until Bun had matured more.

A couple of days ago, I discovered ElysiaJS. Elysia uses Bun as its default runtime and boasts performance figures that seem almost unbelievable. The framework looks so clean and fun that I decided to test it and see if it's worth further exploration.

For my tests, I plan to reimplement one API endpoint from our existing Express.js application. Our current primary challenge is request latency. We primarily focus on delivering sensor data with 'low latency' to user dashboards or apps. A commonly used request in our service typically takes around 700ms (eww..). We've had to delay implementing more features because they would further increase this latency. Therefore, reducing latency would be incredibly beneficial.

The request I plan to use for this benchmark will involve the following steps:

Authenticate the user using a JWT (JSON Web Token).
Make two requests to MongoDB: one to fetch the user's credentials and another to retrieve the devices the user has access to, using Mongoose.
Execute a SQL request for each device using Postgres.js (PostgreSQL).
Convert the data to a format that's more user-friendly for the frontend.
Append additional data from a lookup table.
Return the JSON payload, which is approximately 9kB in size.
All of this will be served behind an Nginx load balancer inside a docker inside our Kubernetes cluster

So let's start to implement the API in bun/elysia. First, we need to install bun and start a bun project. Thanks to "bun create" templates this can be done in seconds.

curl -fsSL https://bun.sh/install | bash  # install bun bun create elysia appcd app

My basic Elysia app looks roughly like this (censored):

import { Elysia } from "elysia";import { swagger } from "@elysiajs/swagger";const config = require('../config/config'); // a config objectimport postgresjs from 'postgres';const mongoose_models = require('./get_mongoose_models');const mongoConnection = await require('./get_mongoose_connection')(config.DB.MONGODB);const Entities = require('./Entities'); // a data service classconst app = new Elysia()  .use(swagger()) // will automatically generate openAPI specification!   .decorate("mongoose_models", mongoose_models(mongoConnection))  // will append the mongoose-models to each request  .decorate("postgresjs", postgresjs({    host: config.DB.MAIN.HOST,    port: config.DB.MAIN.PORT,    database: config.DB.MAIN.DATABASE_NAME,    user: config.DB.MAIN.USERNAME,    password: config.DB.MAIN.PASSWORD,  }))  // will append postgres instance to each object (uses a singleton)  .decorate("config", config)  // append config to reach request  .decorate("entities", new Entities())  // append data service  .get("test", async (req) => {  // add endpoint     // you can use req.mongoose_model and all other decorators here    // HERE WAS MY ENDPOINT CODE (CENSORED)     return { my: "payload" };   })  .onError(({ error }) => {    console.error(error);  })  .listen(3000)

So, what about the performance? To evaluate this, I'm using Autocannon for request benchmarking, comparing it to our existing Express.js application :

That was unexpected!

First of all, I can't quite explain why the original Express.js application performs so poorly with Autocannon. Normally, a single request responds in around 700ms. However, Autocannon records an average response time of about 1535 ms, while Elysia impressively clocks in at 346 ms! That's 4.4 times faster than the Express.js app under Autocannon, and even when compared to the usual 700ms response time, it's still about twice as fast.

Update: For this comparison, I excluded some features from the Express app that weren't present in the Elysia prototype. Notably, the Express app uses a request logger. This may introduce a significant overhead. As I continue to explore Elysia, I'll keep an eye on how the narrative shifts when a 1:1 feature comparison is reached. Should there be notable changes, I plan to update this article.

Initially, I thought a 20% performance improvement would be good enough to justify further investigation, but a 2x increase is definitely a strong incentive for me to dig deeper!

So, what's my takeaway?

Elysia is remarkably fast, approximately 2-4 times quicker than our existing application.
There seems to be an issue with the Express.js app that causes significant performance degradation in Autocannon benchmarks. ;).

]]>

Decoding Plot Data with ChatGPT: A Journey into Chart Analysis

Sven Eliasson — Mon, 20 Nov 2023 20:20:56 GMT

While developing features for our data dashboard, I stumbled upon an unexpected discovery: a customer was experiencing issues with their heating system without being aware of it. It was easy to spot the issue just by looking, but the real challenge was creating an algorithm to do the same. The complexity of crafting such an algorithm was a bit daunting. As a temporary measure, we opted for a human-centric approach, assigning a working student to manually review the data by looking at the data dashboard each week.

Recently, at OpenAI Dev Day, I had a lightbulb moment. The question popped up: Can ChatGPT, famous for working with text, also make sense of charts? Honestly, I had my doubts. It seemed like a stretch, yet interesting enough to try out. To my surprise, ChatGPT was actually not that bad at interpreting chart data.

While I'm still a bit unsure about how useful this is right now, the success of these tests is worth mentioning. It seems like not many in the AI field have talked about this, so I wanted to share my experience. Let's dive into these experiments together and see how ChatGPT can handle graphs and charts in ways that might surprise you.

1) The initial idea: "Tell me what might be wrong here"

This example shows a basic error we're trying to catch. The room's temperature stays almost the same and doesn't match the target temperature, shown as a dotted line in the first chart. The second chart displays the heating power we're sending through the valve. It's at 100% most of the time, but it doesn't seem to raise the room's temperature at all.

Step 1: Add just very few context

"For my first attempt, I simply described what was in the image, to test if ChatGPT could understand anything from it.

Prompt: "This is a plot from a dashboard for a smart thermostat. The first graph is the Temperature with the setpoint in the background as a dottet line, the second graph "Heizleistung" is the "heating power" which is the position of the valve. The third graph is the Setpoint temperature with the actual temperature as a dashed line. The fourth plot it the "flowtemperature", which is the temperature of the water in the heating element. Can you spot a malfunction of the control system?"
ChatGPT: Air Temperature (Lufttemperatur): The room's air temperature remains mostly stable, slightly fluctuating around the setpoint. This graph suggests a well-regulated temperature within the room, staying close to the desired setpoint with minimal variance.
Heating Power (Heizleistung): There is a significant period where the heating power is at or near 100%, which may suggest an attempt to correct a large temperature discrepancy or respond to a significant drop in temperature. However, despite this high heating power, the actual temperature doesn't appear to overshoot the setpoint by a large margin, which is positive. It's worth noting that the dense clustering of dots, especially during the times when the heating power is"

At first glance, this looks like chatGPT completely misses what's actually going on and was about to close this experiment as a "nice try". But one sentence caught my attention: "However, despite this high heating power, the actual temperature doesn't appear to overshoot the setpoint by a large margin." This was a correct observation, just not interpreted correctly. It showed that ChatGPT may have some understanding of the shown data!

Step 2: "Let me tell you what to look for and tell me what's wrong".

Maybe we just need a little prompt tuning? I added a very long explanation of what to look for. Here is the full prompt as a reference:

Prompt: You are a thermostat control system data analyst. I will show you the historic sensor data of a dashboard of a smart thermostat and you try to evaluate if it has potential problems that we need to tell the user. This is a plot from a dashboard for a smart thermostat. 1) The first graph is the (air) temperature in orange, red or green with the setpoint in the background as a dashed gray line 2) the second graph "valve position" also called "heating power" which is the position of the water valve to the heating element with a blue, yellow or red line. 0% means no heating, 100% means all heating available. In the background of this plot is the actual temperature as a gray dashed line. 3) The third graph is the Setpoint temperature that the user selected with a black line and the actual temperature again as a gray dashed line. 4) The fourth plot is the "flowtemperature", which is the temperature of the water in the heating element with a black line. The actual temperature in 1) should somehow follow the setpoint temperature and now over- or undershoot more than 1C during the day. On night times it is expected, that the temperature will not directly follow the drop in the setpoint since the room needs to cool down. Thats okay. The heating power in 2) indicates, that we are currently trying to heat the room. I might be, that the heating system is not responding e.g. in case the water valve to the heating element is closed or malfunctioning. You should see a rise in the temperature in 2) or 1) once the heating power rises. In case the target temperature is not met over a longer period of time while heating power is 100%, there is clearly not enough heat to heat up the room. The flow temperature might be an indicator if the heating boiler system works at all. In case the heating power is set to 100% and we dont see a rise in flow temperature, there is probably a problem with the system. The target overall is to save energy while maintinaing a good temperature during working hours. You carefully remember all of this and review dashboard images accordingly.

And chatGPT's answer:

Based on the image provided of the dashboard for a smart thermostat, here's an analysis of each of the graphs:
Air Temperature: The air temperature (orange line) appears to be relatively stable and follows the setpoint (dashed gray line) closely. There are minimal fluctuations, which is a good sign that the thermostat is maintaining the desired temperature within the expected tolerance. There do not seem to be significant over- or undershoots beyond the 1C threshold during the day. This suggests the system is responding appropriately to the setpoint.
Valve Position/Heating Power: There are times (notably around Tue, 31 Oct) where the valve position (red line) goes to 100%, indicating maximum heating power. During this time, the actual temperature (gray dashed line in the background) does not show a corresponding rise, which could indicate that the room isn't heating up as one would expect when the valve is fully open. This may be a sign that either the heating system is unable to provide enough heat or there's an issue with the valve not actually allowing flow despite indicating it's fully open.
Setpoint Temperature: The setpoint temperature (black line) is adjusted at several points, and it's clear that the actual temperature (gray dashed line) tends to follow these adjustments, though not always immediately. This lag could be normal due to thermal inertia, as you've noted that it's expected for the temperature not to follow the drop during night times immediately.
Flow Temperature: The flow temperature (black line) does show variations, and these should correspond to the valve positions. However, the graph seems to indicate that the flow temperature does drop at times even when the valve position is high (close to 100%). If this happens without a corresponding decrease in valve position, it might indicate a problem with the boiler system not heating the water adequately.
Based on these observations, the main concerns would be:
The periods where the valve position is at 100% but there isn't a corresponding increase in either the air temperature or the flow temperature. The potential lack of responsiveness of the system when the setpoint temperature is changed, which could indicate an issue with system latency or an underperforming heating system. It would be recommended to check the heating system, especially the water valve and the boiler, to ensure they are functioning properly. If the valve is not truly opening or if the boiler is not heating the water adequately, these would need to be addressed to ensure the system can maintain the desired temperature efficiently.

ChatGPT got a lot of observations wrong, but some were correct! The final summary actually described the main problem well. It seems like ChatGPT may interpret charts given a description of what to look at. But there are also so many incorrect observations...

Step 3: "Go step by step, review your observations and finally pick one"

It looks like even after adjusting the prompt, ChatGPT still makes some incorrect observations when given a zero-shot prompt. ChatGPT seems to struggle with understanding concepts like "close enough" or measuring the difference between two lines. In our case, the temperature was stable but not following the setpoint, which ChatGPT failed to consistently detect, even with prompt engineering.

However, the fact that there were also some correct observations is promising. It suggests that with a follow-up prompt, we might be able to guide ChatGPT to focus on and expand upon its accurate observations.

I tuned the initial prompt to make the initial response more detailed and then using a second prompt to ask ChatGPT to focus on the observation it is most confident about is a good strategy. This two-step process could help filter out incorrect interpretations and highlight the AI's more accurate insights.

Prompt after first response: We want to contact the user. Pick the most critical observation and summarize it in at most two sentences. This will be send to the user so it should be correct and professional. Respond in this format: 03.11.2023 10:00 - 03.11.23 23:00 : OBSERVATION
ChatGPT:
03.11.2023 10:00 - 03.11.2023 23:00: Our monitoring indicates that the heating system is frequently operating at full capacity without adequately increasing the room temperature. This may suggest a potential issue with the system's ability to deliver heat efficiently, and we recommend a system check to ensure optimal performance and comfort.

This looks promising, but there's a chance it might not truly understand and just hallucinated answers I might like. It could be making good guesses based on the errors I mentioned in my prompt. The selected time seems to be random, why didnt it pick up the whole time frame from 31st Oct until the end? I will explore the time frame detection in a later post. When providing examples of correct working thermostats it didn't make the same observations. But this isn't solid proof of its understanding and I think I might need to go one step back and test chatGPT's basic skills in understanding.

2) Basic chart data reading

Let's test if ChatGPT can interpret data. I began with a basic time series chart depicting a fluctuating sine wave, which includes an anomaly. This graph represents the temperature in a forest experiencing a nighttime fire. I used this image to explore what ChatGPT can discern from it. Interestingly chatGPT initially refuses to do this! Some motivational pep talk was needed for it to try.

Prompt: This is a screenshot of a sensor dashboard. Carefully look at the graph and the time shown on the axis. You try really hard to be correct and exact. Im under pressure so you have to help me. I know this is hard, but please try. Return a list of time and value that represents this plot. It doesnt need to be perfect. Just as good as possible. Return a value for every hour in a JSON structrue. Do not explain.

I created a plot that compares the values ChatGPT returned with the original data.

Not that bad! ChatGPT did well in some areas, capturing the early part of the graph pretty accurately. It traced the overall trend well, but it totally missed the anomaly later in the chart. Despite trying various prompts, asking for a review, and self-reflection, ChatGPT was unable to identify the anomaly. Overall, it's fair to conclude that ChatGPT can extract values from a chart, though with limited accuracy and it seems to follow general trends. Improving the chart's design might enhance this accuracy. I plan to delve into this in a future article. (Spoiler alert: initial tests suggest that tweaking the chart doesn't significantly improve accuracy.)

In the first part of my experiments, I found that ChatGPT can read and understand data from a chart, but its accuracy is limited. Next, I plan to explore how ChatGPT processes chart information. Although this method currently has limited practical use and there are likely better approaches, like LSTM or parsing data JSON encoded instead of visually. However, it's an interesting topic to investigate and who knows how ChatGPT will evolve in the future.

In the second part of this series, I'll conduct more experiments to provide a more quantitative analysis of ChatGPT's strengths and weaknesses reading chart data.

Observations so far

ChatGPT can read data from a chart with limited accuracy
ChatGPT can correlate two distinct plots and compare their relative trend.
ChatGPT struggles to merge visual proximity with numerical similarity. It often fails to grasp the differences between two lines on a chart. Additionally, when we zoom in on the same data, ChatGPT's interpretations change. (More in part 2)
ChatGPT can identify the timing of events observed in data, but its consistency is not always reliable. (More in part 2)

Update on Part 2: GPT became more restrictive and simply refuses to readout the data from the chart more and more. Need to investigate a reliable jailbreak before part 2.

]]>