From 49da941b059a284386b55f80d16cf0c6d8556600 Mon Sep 17 00:00:00 2001 From: simonox Date: Tue, 26 Sep 2023 21:11:36 +0200 Subject: [PATCH] presentation --- docs/workshop/README.html | 208 ++++++++++++++++++++++++++++++++ docs/workshop/README.md | 54 ++++++--- docs/workshop/images/qrcode.png | Bin 0 -> 5923 bytes 3 files changed, 246 insertions(+), 16 deletions(-) create mode 100644 docs/workshop/README.html create mode 100644 docs/workshop/images/qrcode.png diff --git a/docs/workshop/README.html b/docs/workshop/README.html new file mode 100644 index 0000000..6760f04 --- /dev/null +++ b/docs/workshop/README.html @@ -0,0 +1,208 @@ +
+

Fab City Dashboard

+

This is a prototype of data management platform for Fab City Hamburg based on existing open source solutions.

+

width:100pxy

+
+
+

Features

+
    +
  • Eclipse Mosquitto: MQTT broker
    • +
  • Node-RED: Flow-based development tool for visual programming and data flow automation
    • +
  • InfluxDB: High-performance data storage
    • +
  • Grafana: Real-time data visualization and monitoring dashboard
    • +
+
+
+

Today's Goal

+
    +
  • install the stack
    • +
  • configure one or two data sources
    • +
  • transform data and store them in a database
    • +
  • visualize data stored in the database
    • +
+
+
+

Architecture

+
    +
  • +

    Data is collected by IoT devices, e.g. an ESP32 based power monitor. These devices publish their data via MQTT into a topic in a message broker. We use Eclipse Mosquitto as a MQTT message broker.

    +
    • +
  • +

    Node-RED is used to read and transform or combine data and to implement more sophicsticated use cases like notifications or worksflow. Node-RED subscribes to topics in Mosquitto and can be used to save transformed data into a database.

    +
    • +
  • +

    As our data is bases on time, we are using a time series database to store information. We used InfluxDB as this database.

    +
    • +
  • +

    Dashboards can already be created in Node-RED, but to be more flexible (and include more options) we are using Grafana. Grafana reads data from our database and other sources (like CSV files on the Internet) and displays them in a nice dashboard.

    +
    • +
+
+
+

Sensor level: Physical acquisition of measured values

+

Sensors can be build or bought:

+
    +
  • +

    Sensor head

    +
      +
    • ESP32 or equivalent
      • +
    • OpenHardware or COTS (commercially of the self)
      • +
    • Analog/Digital acquisition
      • +
    • Communication gateway via WLAN / MQTT
      • +
    +
    • +
  • +

    Examples measurements

    +
      +
    • Energy measurement
      • +
    • Temperature sensor -> temperature measurement
      • +
    • Pulse sensor -> incremental measurement of filament
      • +
    +
    • +
+
+
+

Example sensor - Open Engergy Monitor

+

[bg contain]

+
+
+

Example sensor - Open Engergy Monitor

+
+
+

Example sensor - Shelly Plug

+
+
+

Example sensor - GCode Sender

+
+
+

MQTT (Message Queuing Telemetry Transport)

+
    +
  • publish-subscribe, machine to machine network protocol
    • +
  • designed for connections with remote locations +
      +
    • devices with resource constraints
      • +
    • limited network bandwidth
      • +
    • Internet of Things (IoT)
      • +
    +
    • +
  • runs on top of TCP/IP, QUIC (UDP) or Bluetooth
    • +
+
+
+

MQTT - Components

+
    +
  • one message broker (Mosquito) and many clients
    • +
  • broker receives published messages from clients
    • +
  • routes them to subcribed clients
    • +
  • clients subscribe to topic patterns
    • +
+
+
+

Flow - Node-RED

+

The idea behind it is to make it very easy to connect APIs, hardware devices, and anything else accessible over some type of network connection.

+ +
    +
  • open-source
    • +
  • low-code
    • +
  • visual programming tool
    • +
  • flow-based development
    • +
+
+
+

Node-RED - Core Concepts

+

Nodes are the important part of Node-RED, they:

+
    +
  • are triggered by either receiving a message object from a previous node or an external event like an MQTT event
    • +
  • process messages or events and then passes them on to the next node
    • +
+

A node can:

+
    +
  • Inject: Starts a flow by injecting a message or a payload.
    • +
  • Change: Here you can do basic transformation or modification on the message object.
    • +
  • Debug: Can be used to help developing flows by sending messages to the side bar.
    • +
  • Switch: Here you can add logic (like sending the message to different nodes).
    • +
  • Function: Add custom JavaScript for uses cases where simple nodes do not do the trick.
    • +
+
+
+

Node-RED - Plugins

+

Node-RED uses plugins:

+
    +
  • extend functionality (like dashboard)
    • +
  • connectors (like influxdb)
    • +
  • libraries (like aggregating watts and traosnform them to khw)
    • +
+
+
+

Node-RED - Simple Flow

+

+
+
+

Database - InfluxDB

+

InfluxDB is a database for any time series data. Time series data is everywhere, since time is a constituent of everything that is observable. As our world gets increasingly instrumented, sensors and systems are constantly emitting a relentless stream of time series data. For example:

+
    +
  • Electrical activity in the brain
    • +
  • Rainfall measurements
    • +
  • Monthly subscribers
    • +
  • Heartbeats per minute
    • +
  • Electricity consumed by a chain saw
    • +
+
+
+

Database - InfluxDB

+

+
+
+

Dashboard - Grafana

+

Grafana is an open source analytics and interactive visualization tool.

+
    +
  • charts
    • +
  • graphs
    • +
  • alerts for the web when connected to supported data sources.
    • +
+

As a visualization tool, Grafana is a popular component in monitoring stacks, often used in combination with time series databases such as InfluxDB.

+
+
+

Dashboard - Example

+
+
+

Let's start

+ +
+
+

🫵 Hands on

+
+
+

Where can I find this presentation?

+ +
+
\ No newline at end of file diff --git a/docs/workshop/README.md b/docs/workshop/README.md index 22601a8..2020406 100644 --- a/docs/workshop/README.md +++ b/docs/workshop/README.md @@ -1,12 +1,13 @@ --- marp: true +class: invert --- # Fab City Dashboard This is a prototype of data management platform for Fab City Hamburg based on existing open source solutions. -![](https://code.curious.bio/curious.bio/iot-backend/raw/branch/main/docs/workshop/images/architecure.svg) +![width:100pxy](https://code.curious.bio/curious.bio/iot-backend/raw/branch/main/docs/workshop/images/architecure.svg) --- @@ -59,20 +60,26 @@ Sensors can be build or bought: --- -# Example sensor - Open Engergy Montior +# Example sensor - Open Engergy Monitor -![](https://code.curious.bio/curious.bio/smart-energy-monitor/media/branch/main/docs/images/breadboard.png) -![](https://code.curious.bio/curious.bio/smart-energy-monitor/media/branch/main/docs/images/clamp1.jpeg) +![[bg contain]](https://code.curious.bio/curious.bio/smart-energy-monitor/media/branch/main/docs/images/breadboard.png) + +--- + +# Example sensor - Open Engergy Monitor + +![bg left](https://code.curious.bio/curious.bio/smart-energy-monitor/media/branch/main/docs/images/clamp1.jpeg) --- # Example sensor - Shelly Plug -![](https://code.curious.bio/curious.bio/smart-energy-monitor/media/branch/main/docs/images/shelly_plug_s_1-1.jpg) + +![bg right](https://code.curious.bio/curious.bio/smart-energy-monitor/media/branch/main/docs/images/shelly_plug_s_1-1.jpg) --- # Example sensor - GCode Sender -![](https://pad.fabcity.hamburg/uploads/586181bc-6de2-46b8-9eba-8a11fbb59a5f.png) +![bg left](https://pad.fabcity.hamburg/uploads/586181bc-6de2-46b8-9eba-8a11fbb59a5f.png) --- @@ -95,7 +102,7 @@ Sensors can be build or bought: * routes them to subcribed clients * clients subscribe to topic patterns -![](https://upload.wikimedia.org/wikipedia/commons/8/82/MQTT_protocol_example_without_QoS.svg) +![bg right](https://upload.wikimedia.org/wikipedia/commons/8/82/MQTT_protocol_example_without_QoS.svg) --- @@ -104,7 +111,7 @@ Sensors can be build or bought: The idea behind it is to make it very easy to connect APIs, hardware devices, and anything else accessible over some type of network connection. -![](https://nodered.org/images/nr-image-1.png) +![bg left](https://nodered.org/images/nr-image-1.png) * open-source * low-code * visual programming tool @@ -114,10 +121,10 @@ The idea behind it is to make it very easy to connect APIs, hardware devices, an # Node-RED - Core Concepts -Nodes are the important part of Node-Red. +Nodes are the important part of Node-RED, they: * are triggered by either receiving a message object from a previous node or an external event like an MQTT event -* processes the message or event and then passes it on to the next node +* process messages or events and then passes them on to the next node A node can: @@ -147,9 +154,19 @@ Node-RED uses plugins: # Database - InfluxDB -InfluxDB is a database for any time series data. +InfluxDB is a database for any time series data. Time series data is everywhere, since time is a constituent of everything that is observable. As our world gets increasingly instrumented, sensors and systems are constantly emitting a relentless stream of time series data. For example: -![](https://code.curious.bio/curious.bio/iot-backend/media/branch/main/docs/flow/docs/images/influx-data-explorer.png) +* Electrical activity in the brain +* Rainfall measurements +* Monthly subscribers +* Heartbeats per minute +* Electricity consumed by a chain saw + +---- + +# Database - InfluxDB + +![height:600](https://code.curious.bio/curious.bio/iot-backend/media/branch/main/docs/flow/docs/images/influx-data-explorer.png) --- @@ -167,23 +184,28 @@ As a visualization tool, Grafana is a popular component in monitoring stacks, of # Dashboard - Example -![](https://code.curious.bio/curious.bio/iot-backend/raw/branch/main/docs/workshop/images/sampledashboard.png) +![bg left](https://code.curious.bio/curious.bio/iot-backend/raw/branch/main/docs/workshop/images/sampledashboard.png) --- # Let's start -* Check installation of +* Check installation requirements * 64bit environment (AMD64, ARM64) * Docker * Docker-Compose * Checkout the repository: * https://code.curious.bio/curious.bio/iot-backend -* Let's follow the README, together +* Let's follow the README, together! + +--- +# 🫵 Hands on --- # Where can I find this presentation? -* https://code.curious.bio/curious.bio/iot-backend/src/branch/main/docs/workshop \ No newline at end of file +* https://code.curious.bio/curious.bio/iot-backend/src/branch/main/docs/workshop + +![bg left](https://code.curious.bio/curious.bio/iot-backend/raw/branch/main/docs/workshop/images/qrcode.png) \ No newline at end of file diff --git a/docs/workshop/images/qrcode.png b/docs/workshop/images/qrcode.png new file mode 100644 index 0000000000000000000000000000000000000000..43d41d8063703da5ea55d5ce92ad14de6fa082f3 GIT binary patch literal 5923 zcmZWr2Rz$d_YbX2+EOb>Rclm35L+TNX6?O7?H#ejtW~3gDlKYD?OD5a%^IbuXw9Ou z_A07`fBL-d`@GNR{pa(!x#!+*)nau$G*ci%_eOuBBY5RS)|m2T7A}Y{LDU>WQDFR;%FzZkKT;ieV3~ z(TqD|1J+Ta!V=?mT@xO&<~?hpe+-V7vZwlf%vhkLayIawPyxv+1$R!U^-qfXE(sGG zek9w6T4H{3kNd>yM0We&G(gQ+YxJzr0<2dfephPNNLR*1_K{$Ya8)|1zyPlM@aN0K z*rz=-HM87-R3F2+_hvNVmtZ=N2wL~uK#b?wt++gv3r52aHfFYKf@20ZOG ze=cSosT^fl|1>ovQ_+UADo)L2|BF41lOdDgBtX4m;!D}Cp!X`>y(iySgIUGiF`kFE z#%3L~N74gKsR7y{k~`yb6aelUp{u|Kl{CZ#O6!qOC?%g9xEMf}dx(`j}NCbnmo@_Muq4HoKQXh^<(Q?EFaYpii?kV`ZA+@uQr)C zpfozETeqqC56Z=H;qTudhK^wg==! z-O69>e~|`pY)AC&s$tp5v~~FuQXA3S2VzbbkN0upEy!j z4yg}Waf*6lqIGwsskY)0LcTauC(H0E#xhXEzC2K=>8*1ir34M?3!c_ z;a+`Yi^ffm2PBp3ewY2gQ*mrv!Ch*LVta;IOb{eHakM*i>B{oxV&Go)FRE}_`NXd( zlw9tVWmGd%Vv*uq0$o?F!uixhIi(r8WAF6fdPJ;m?a}WE?`5T=q%@>N=*L{wa*ohX z|0G;;SND=`j;iNvn`abv5^i&$Z*g$dCxf5PCk@yJ*RV;w$W)wHs^oom8_E6XS>gQ* zRak-4$Blx!1sMf_1rPNouOp}Y$f)=UiK0*w-0v|y^&L#E?&3l zqw)&bpxfYsm3u2ZE3zw4eo;o~bN%O8`~v)1`eUVir8i6O8XT8c=myuzg@lch9))+2 z^)tiuBPQY}?oCVJcS zm2MH+C`35e9BP6|v`C#-4!%U-BfS4D@cS0|Ah~)3djxqzf0QlCsGBirBFYa}t>lf@ z$G0gp;;d}UmTi~sFPrgH$7WHD+E%HK-W$Fb{rK$h1kI_2Y9Z~o>p1y12~4pTPz$eR zs-0SGUVXhP^S))c!{oE+E0a&g@S2p`pS8oauC+lfcCNh*rFG3N8VyGc7MM9_LSJ7` zluPaCYP-OM_y^xwbo7>yOrT7jIexCVtunYZ_<1nWJR7XOaKB?I)WhA+c*nhOu@;|Y z^J3YfzavvUOE(KBgM)||_n7O~I({&jNnZ)-G}*l873+8MMRBuqWze(2H+Z#hxzD4_ zhs~ZvSeHearD{%K_VISnEHXecp#0GBSi%1{mMOqDaKK*?Yk)-@P9ACQyX}GxULTq) zk8j8A9UjlE&5Y_h)TAE{3nK$N>O->5Z(j&qP+vHZ^pSLhHpoeSfVvxfUG5EtJBc$3 zm%9pcH2RQi8S$0zZO3CvR5*$sl^tywwHoF7Sf~3%GzakX>NdC?ai$Y#70H)>>lr;O zv&oGU;XzobC(^>=4BkB>o9~+f98A3mfIC zlH4H_7HgnS)=${>DIV>fE<`cBZ*;2EEXDsU@Ogd0ulW%cr-dSE@Inh15?h^}~H?wfP;3_EYB) zjkR+`4~|`?-dJvw>Nu?Dop$*4+1oDvyre=lO((A*+O- zi_p=qRcFHb$B54nUnp{>6ehh1_bv+B4eJ|h9Q1+*vQ_vM=`cIWSi9 zjB64rtp8rOu$CZR)S%aqPVoH^KbbyQOYP8@b1QoYDd8jg{S4cZOPF~4iY9_%NCs>k z-;QmBy{bKZc2RIQwGH}k)7m?$V<~85*r30xsA{aL{y}5Qd{FmMUr9!TgqF{YkHkVv z$3(!)qOaL#ezsRzWgGK0cK%IJs{e7o!cu#7;I-rJc8sr0})S#%KU% zKR~XH4&X813Psp_aJ@jHz9RQrI}kR$0QOA^&LBDfi3AB>m7J-GLJmEJoDkWb49Vwd zQ<2r*^XcH=X$Xi!r97iBdP!$jD`RV+W9T#omS$K_a*=sKCSywOwMy%%LVUX#+8L|b zYiI!Yi8LjEB-#mZnMjck2Z%TTfJ@J!02IWTo;Vb8N&n`)%)RtCo%u&lUJs?JN}Tm< zJnih7n#v0n3D2%5ar?{Yypb(cVH76%0+|$-xT312o-*Dmz z!R6@f?Jf<4`uX_@`oRP-o(@o9DJdzakO)*nM1UwE;N|b;ZG{zZ^SbpfCI8W*VCQAy z>E!P1gmL5iqu0tBWzEzx+B0IZGn zO6}e)n+jBPJsd1K!W5Pk^Lm}5-e6Y&UJD0hP0}HtUUImGVN((_iZ1*( z4=zN!YIaNuo5Bn7TYGmVk}wtkx9D43u|2AV!ww5U0(uV&fJ_-;@7LwkTc)yM?6kGi z6o%zza z?!R+-W5Us7+)z~2P)!j$sTcMJgu;8(cXwTEg@QD2)xBX=ELn_xdDQWx42l9dLihD> zNBUOr%oT!vsfc}u8S|B z`cG!N{Oa_|OPetImZC_0cYYyFsf=vkv?H->NHW{|ommmH)bXnfOY!TLEI*K*U(fY+ zKv`N+7j~`Abx@hzqQL?@>_8Pm_IcEo3Evgtyv^TAok<&oNR8?6W5-Lrh!cNF_bbxD zr87&_f`h3$u5AWMwye33yg}5`e4#i?)FQpfs}|h*V>n@m7l(|MS-vr}X3o)tvzKR4 z?URiAvd>zHvpS8JNbYv`EjOt!k&U}{_-?r-&BE38As)8$%({?RsTyW{p)r@7>tfWN zr2BFPDaQ?;tHc|?K^U*H938bojup)vpvAYQ@sDOXy}7t39@sM#b**1>;H7EFhW@%Q z>__pEtnpEClJP^8RzXqB!jSWUQBW3d&PZHc$(V(A8gJ@7WruIc;Mjrd2~EEGUoFTx zRx+b(HssI^=l8c>c=f~3b&%I$WOO!Z;_X|?o?MdRQzV&Wv{i#vmk64D^{{ictBQLnr3;ME!>l&&E}hHto5 z*cJN3#p82r?}&AF_}ldi^V)i`a-Bp`X$LL2lbggUxS4TXbR!gmo|IpSX8(19IXd6x z`U3A0Wm;b3iei4$Kh|v})rKt$v#M*-Vee`0uJ-`<3(}US?j6;w$JuKcj1IAD;{|FB zJ10rRUizSjOO*=*C}RxI&l;6=;F{xMnb~W(*A#VvMuvy^TLAm;2?jgyQ{#hmLhMZa zllO2@`P-D}S#w&3*Pid42ls{IOMkN*#f+%57uAz?kC%_FNxdS_6A#KQPTk`bnxMI( zw-Gk7Y2@XwCj=F-aGguT9BGR{}G>zlcHhwmnE}Fous_drvQJ_!|B#5vwx)I)I7#@FR&-+EK3+s;pnJfbY*AI zp-DV(>*1SQ9Zbn9{-+QJZw=p5tk`#!<1JkW{+a`6}i{ zQrbj8#fPVEqC2Fl*Wy@`*hN`|-E*u0q+<5I$vE2`ZhmyAk6-`rW7g;X!x6mF=|1^q zgOC94gb-;Hc}5o6DKnc^UkLUDGvg3a4Z?dsB<37x%dIPZce2jMTQw*}p^x`imtoRe zEG!YyTstvIW6u7f(iLn07OTX3rq`1;0NEv^DD9;v_ftyo6uIi z&eQ?NJEXlcBY@pR?;dFuGus@O(U`>cdoTD`>r!+pp=Eq-N{steIC)!W{_rayI#_T- zedZ~L^Oee5C;++=8mSi56K8paDCR6#(%E||I^~9o=G#oUn$t$prk2N;>E=G3{>ht~ zDxh}gn8<5f8GeZDt zVxSR816Cj<4}SQr&}Z54Y?IoFdYT{w6hEkq-gq|Znr;w2#apg#ZU0Wen z;JRIirF!6wRULs+lT2wjU`b0W@J9j@BRV#%#J9t$CXLTX6e#7lClXw2r%X^+wyiKN zQ*!9>=~b7>Rmy1423#XH2H1Z3v2`nfN0Of5&gPTYB`fxX1e&VjI2M+Sw+iuE!9}>J zC@lFrBgTUzY7WB!A%87QactBI8;O}TH#OW<5K4#o`Gd0jMjYFS{mZl@i?oHwlZZ!p#ySbm-s z!Gf|_r&3^>#EgCx;kIR`OdO&8<*Mm@HXJhP;A3&3(`M9qY=)a!MhX7Be$2|W`k7_g zvF&n(Yfc+GX2JSBTp7z%_};O}Q)h_Z;@g{FJ0V7EP0KFCFDSyV&WKnj6~p&Q4%u_@ z2eDck+@9XFb?{FaZ<~JQocEe49af()oaN%gO=m~2qBz!#HTih;ATkVj_jtg1yVCa( z-UAVH=eksmA0;crW15Kc)HUjbF{K@`YdP7?c%gZVV2G^ljXnkU0BKNm^x_J>a08Ly zw-VU#JKrDZ)X_r>PAraek9ddvT88a%ysz;;zRc#5b9bA>#I3A5uyU!m>r27Za{3o7 zv1!`H46iMuO3TYlRV2DC8;H{!y}6Z2d}G{wt;3mpsp>|SjIyzqky!Q=?bJE-Jf4ki z2Vg+Xxox+5xUtpP6!U^|q*v--V^5)OWLMc!O!}7K9s6k#vFW65;J<}k1*j;Dg9ZT!3l28}UHw;P05Wb!7&*&C8 z(}$Jbbw_XiZ+ppCk~#6Vc!fJZmVT(8b6={5BzO!c71KDRjc+4FUMbPS`M#5HHID*q zZSe-F1UuKEfJ<8