Vereinschronik in einem Content Management System … schlechte Idee!

Als Chronist in meinem Musikverein ist es meine Aufgabe, die Ereignisse und Termine, ob nun musikalischer Art oder außermusikalischer Art, zeitlich (chronologisch) geordnet niederzuschreiben und einmal im Jahr einen Bericht zu halten.

Um Personen Einblicke in diese Chronik zu geben, und auch, um die Chronik der Nachwelt zu erhalten, wurden anfangs Bücher verwendet, in der die Chronisten handschriftlich ihre Berichte verfassten, und dann noch Bilder oder Zeitungsartikel einklebten.

Mit dem Siegeszug der Personal Computer und der Textverarbeitung ging man dann Ende des 20. Jahrhunderts dazu über, die Berichte am Computer zu schreiben, digitalisierte Bilder einzufügen und dann ein oder mehrere Jahre in Gesamtheit auszudrucken, und als Buch binden zu lassen. Vorbei waren die Zeiten des Tintenkillers und der verklebten Seiten.

So ab 2005 begann ich dann, die Chronik des Musikvereins zu digitalisieren: Es gab kein physikalisches Chronik-Buch mehr, stattdessen war die Chronik ab diesem Zeitpunkt öffentlich einsehbar in einem sogenannten Content-Management-System (CMS) im Internet. Alles ganz praktisch, die einzelnen Ereignisse oder Termine konnte man online als Artikel der Kategorie Vereinschronik verfassen und bei Bedarf ändern, später dann auch noch Fotos hinzufügen. Und die zeitlich geordnete Liste dieser Artikel war dann quasi die Vereinschronik.
Eigentlich doch sehr elegant, oder?

Wir befinden uns im Jahr 2022, rund 17 Jahre Vereinschronik befinden sich im CMS. Das CMS läuft (aus noch unbekannten Gründen) nicht mehr rund, da müsste eigentlich jemand regelmäßig Updates einspielen, auch wegen Security und so, und überhaupt, was ist, wenn mich ein Bus überfährt? Hat jemand überhaupt eine Chance, noch an die Daten zu kommen, oder sind diese 17 Jahre für die Nachwelt verloren?

Wie lange wird es dieses Content Management System (in meinem Fall: Joomla) noch geben?

Und warum kann man keine Videos zu den Berichten hinzufügen? Und warum muss man die Bilder immer manuell auf das Format 640×480 runterskalieren, bevor man sie hochladen kann, wo heutzutage doch eigentlich Bildformate wie 3840×2160 Pixel der Standard sind?

Aber es kommt noch schlimmer: Im Jahr 2005 erschien es noch eine edle und gute Idee, die Vereinschronik allen Interessierten öffentlich bereit zu stellen, um sich ein Bild über den Verein zu machen. Doch die Zeiten änderten sich. Darfst du ohne Genehmigung einen Artikel aus der Online-Ausgabe deiner örtlichen Zeitung übernehmen? Wie ist das mit Bildern aus der Presse? Darfst du bei Ständchen für ein Mitglied den Geburtstag und das Alter des Mitglieds nennen, und vielleicht sogar die Adresse? Und bei selbst aufgenommen Fotos: Ob die Leute auf dem Bild wohl damit einverstanden sind, dass man ihre Bilder öffentlich sieht? Und ich rede noch nicht mal von Kindern …

Kurzum, die öffentliche Online-Chronik war zu einem rechtlichen Minenfeld geworden, und war nur noch für mich selbst online zugänglich. Damit fiel aber auch ein Hauptargument weg, die Chronik in einem CMS zu speichern.

Was im Jahr 2005 als gute Idee erschien, entwickelte sich nach und nach zur Altlast.

Der Schritt war also mehr als überfällig, jetzt (2022) die Berichte und Fotos aus dem Internet wieder auf die lokale Festplatte zu holen, und den Einsatz des CMS generell in Frage zu stellen.

Festplatte heißt in meinem Fall, ein Netzwerkspeicher (NAS) in RAID 1 Konfiguration: Selbst wenn eine Festplatte den Geist aufgibt, sind die Daten noch redundant auf einer zweiten Festplatte. Es gibt keinen Datenverlust.

Aber wie kopieren vom CMS auf die lokale Festplatte? Das sind rund 480 Artikel in den 17 Jahren, das geht nicht manuell! Mein Vorgehen war das folgende:

  • Ein Backup der Joomla Datenbank ziehen, und zwar im JSON Format.
  • Manuell alle Einträge in der Backup-Datei löschen, die nicht mit mit den eigentlichen Artikeln zu tun hat (Datenbanktabelle Articles).
  • Kleines Python Script schreiben, das alle Artikel dieser Backup-Datei nacheinander durchgeht, und dann das Datum im Titel verwendet, um den Dateinamen zu bauen. Dabei Zeichen ersetzen, die in Dateinamen nicht erlaubt sind. Und dann ein Verzeichnis mit diesem Namen auf der Festplatte anlegen.
  • In diesem Verzeichnis den eigentlichen Bericht anlegen, mit Dateinamen index.md.
  • Ach ja, MD steht für Markdown, eine Auszeichnungssprache für Dokumente, die auch ohne Textverarbeitungsprogramm wie MS Word in einem beliebigen Editor verwendet werden kann. Schnell zu erlernen, und zugegebenermaßen auch eingeschränkt, aber wir wollen ja auch keine Doktorarbeiten schreiben. Auf jeden Fall ist es sehr wahrscheinlich, dass Dateien dieser Art auch in den nächsten 20 Jahren noch gelesen und dargestellt werden können. Bei MS Word wäre ich mir da nicht so sicher.
  • Im CMS konnte man HTML Elemente benutzen, um das Dokument zu strukturieren oder einzelne Begriffe hervorzuheben etc. Das konnte ich unmöglich manuell nach Markdown konvertieren. Die Python Bibliothek html2text hat dabei sehr gute Dienste geleistet. Sogar HTML-Tabellen wurden oft in korrekte Markdown-Tabellen konvertiert.
  • Wo man gerade schon am konvertieren war, stellte sich die Frage, ob man nicht zu jedem Artikel noch Metadaten speichern will, wie einen Titel, die Namen der Personen, um die es vielleicht im Artikel geht, und den Ort der Veranstaltung.
    Es gab verschiedene Optionen, und ich habe mich entschlossen, die Metadaten im Kopf des selben Artikels zu speichern. Das Format nennt sich YAML Front Matter. Das ganze ist (im Laufe der Zeit) beliebig erweiterbar

Fazit:
Die Vereinschronik ist jetzt in einem Format abgelegt, das zwar nicht online bearbeitet werden kann, aber auf jeden Fall die nächsten 50 Jahre übersteht. Eben aus dem Grund, dass die Artikel in ganz einfachen Verzeichnisstrukturen auf der Festplatte abgelegt sind, und als Auszeichnungssprache Markdown verwendet wird.

Die Bildformate sind nicht mehr beschränkt, können beliebig groß sein. Auch Videodateien sind kein Problem mehr.

Der Weg ist offen für neue Features wie Suchfunktionen, ausgefeilten Statistiken sowie vielleicht Gesichtserkennung auf den Bildern mittels KI. Mal schauen …



Advertisement

Robotics: Optical sensor

The previous article mentioned an Optical Sensor, but mentioned not a single word about that device.

As you remember, I reported that my robotic vehicle with mecanum wheels can not drive in a straight line (Video), on one side due to missing precision in the hardware and manufacturing of the vehicle, but also on simple physical facts like different surface friction of the individal wheels. This may not be a problem for rubber tires, but with this mecanum wheels with spinning 45 degree oriented wheels on the outline of the wheel, everything can happen.

Mecanum wheel

Fail: Control the revolution rates with PID controllers
Actually, the first idea to fix the problem of the vehicle going off the straight path was to make sure that all 4 wheels exactly have the same revolution rate. The gear motors used in this vehicle also have rotary encoders attached, which enables measuring the revolution rate with the microcontroller. The Arduino Mega2560 has more than enough input pins and processing power to deal with the incoming signals from all 4 rotary encoders of the motors. Going back what I learned during university years ago about “control theory” (German: “Regelungstechnik”) I created 4 PID controllers to make sure all 4 motors have the same revolution rate. But that was not successful for several reasons:

  • I expected constant revolution rates per motor, but there where considerable variations in these revolution rates. Possibly this is a quality problem of the motors and integrated gears.
  • The gears do not react very good on abrupt load changes (German: “Lastwechsel”)
  • This does not solve the problem with the different surface friction of the indivual wheels.

Maybe, not so bad at all, because I learned a lot about PID controllers and filters, and how to implement them in Arduino.

Next attempt: Optical sensor
To make the vehicle stay on the desired straight path, I needed some additional sensor input that tells in which direction the vehicle goes, and how much the vehicle has deviated from the desired path. Maybe then some controller can take action and somehow steer in the opposite direction (not tried out yet).

Idea: Why not attach something similar to an optical sensor in a computer mouse to the vehicle?

I found an appropriate optical sensor on tindie.com:
This sensor breakout board offers all the needed components to implement the Pixart PMW3389DM-T3QU high end gaming sensor.
It has a simple to use SPI interface and provides a revolution of 16.000 pixels per inch.

PWM3389 Sensor Breakout Board

That was the point where the fun begun: Although the breakout board already had an appropriate lense attached underneath it, including the optics for the infrared LED that illuminates the surface, you have to make up your mind by yourself how you can position the lense (more or less) exactly 2.1mm above the surface.

In the end, found out that I do not only need some kind of enclosure with hole in the baseplate for the breakout board, but also some kind of mechanism that presses this enclosure gently to the surface, in order to make sure that it does not loose contact while driving (mecanum wheels go up and down).

So, in the end I constructed an upside down fork like on mountain bikes or motorcycles

And yes – it worked!
In the test drive (video) I could make the drive stop after 2 meters, based on x-values received from the sensor, and received a path deviation of 38cm on the y-axis. I plotted out the sensor values for x (red line) and y (blue line) during the drive in Arduino’s Serial Plotter.
Sample rate: 100ms

Output on Arduino’s Serial Plotter

Is this deviation in y-axis enough to have a controller steer in the opposite direction? Or do we need a yaw-rate (rotation around the z-axis) instead?
Stay tuned for upcoming articles.

Robotics: Mecanum Wheel Vehicle and Optical Sensor

This is the first of a series of blog posts about robotics.

Just before the christmas holidays I ordered myself a rather inexpensive (compared to alternatives) robotic vehicle in order to get experience in the hardware of robotics as well as learn about the famous Robot Operating System ROS.

I chose a vehicle with metal chassis and mecanum wheels, which promised a lot of fun because it can go into several directions by controlling each of the four dc-motors separately.

The model is a Metal Chassis Mecanum Wheel Robotic (for Arduino Mega2560) Introduction (Model ZZ012318MC) from osoyoo which I ordered at Amazon for €129.

The good points:
For your money you get everything you need to play and learn as beginner, including rechargable batteries, charger, and nice extra equiment like an ultrasonic sensor, optical tracking sensor module and a servo motor. There are several different lessons you can do with this equiment and in this way learn about robotics and programming in the Arduino IDE.

The suboptimal points:
In the model I received, the four gear motors have non-neglectable difference in their performance, meaning, if the same power is applied to them, the number of resulting revolutions of the wheels are not the same. You can measure that with the attached rotary encoder. Additionally, the mecanum wheels feel to losly fixed on the axis, and the mounting points for the motors on the chassis are not precise enough (Tolerances of about 2mm).
All of this leads, at least for the model I received, to the inability to drive in a straight line (by powering all motors with the same voltrage) for even 1 meter.
Other robotic vehicles like TurtleBot (with rubber wheels and stepper motor) should not have this difficulties in driving in a straight line and are designed from the beginning to work with ROS.

A video of trying to drive in a straight line is here: https://youtube.com/shorts/jt1zlkfRW5g?feature=share

LED strip controlled by WLED

List of parts used

PartShopPriceComment
1m 5m Address SK6812 RGBW RGBWW led streifen Neutral weiß WWA Wasserdichte 30/60/ 144 leds/pixles/m 5V ähnliche WS2812B
Product properties: 5m 30 IP20 + RGBW + Black PCB
AliExpress€18.25
Elektrische Drähte RGB LED Anschluss Kabel 5m 10m 100m 20AWG 5 Pin Elektrische Kabel Draht Für 5050 3528 WS2812B RGBW Streifen Led-treiber
Product properties: 3pin + 22AWG + 5m
AliExpress€8,50In principle you can use any kind of wire, but this one uses the same color scheme like the LED stripe: White (GND), Green (Signal), Red (+5V)
10 stücke 2pin 3pin 4pin 5pin LED Streifen Stecker Für RGB RGBW RGBWW 2812 3528 5050 LED Streifen Draht Verbindung terminal Stecker
Product properties: LED Strip to Wire + For 10mm 3pin IP20
AliExpress€5,05Very error prone to build connetion cables this way. Test carefully. Consider to solder instead to use this connectors.
ESP8266 ESP-12 ESP12 WeMos D1 Mini Modul Wemos D1 Mini WiFi Entwicklung Bord Micro USB 3,3 V Basierend Auf ESP-8266EX 11 DigAliExpress€1,91
AC DC 12V 5V 6V 8V 9V 10V 12V 13V 14V 15V 24V Netzteil Adapter 1A 2A 3A 5A 6A 8A 220V Zu 12V Netzteil adapter Led-treiber
Product properties: 5V + EU Plug + 5A
AliExpress€7,46Depends on how many LEDs this power supply has to drive.
Half-Sized Breadboard, Connection Cables, …exp-tech.de€5As you like
RAD LXZ 6,3/1K0
ELKO, 1000µF, 6,3V, 105°C
reichelt.de€0,23
SET: LED Profil, 100cm Profil LED für LED Streifen, aluminium led profil LT4 + Abdeckung (Schwarz Milchig) [Energieklasse A+]
Marke: LumenTEC
Amazon.de€4,39/mSelect the one that fits your needs

Espruino Pico: Low power ultrasonic sensor with LoRaWAN

Simple LoRaWAN node connected to The Things Network (TTN) using Espruino Pico and Microchip RN2483 LoRa Technology Module. As sensor a MB1242 I2CXL-MaxSonar-EZ4 is used, that is connected via I2C. It returns the distance in centimeter as an array of 2 bytes.

The most prominent goal was to find out how to design a device and its software in order to minimize the energy consumption in order to maximize battery life.

Motivation

  • In many cases, LoRaWAN nodes are deployed in remote places without a static power supply. So both hardware and software need to be designed in a way that allows battery operation.
  • With battery operation, operation of at least 1 year should be technically feasible these days, given that sending sensor values only a few times per day is good enough for the given use case.
  • The Espruino Pico already has some operational modes for power saving, the goal is to find out how these modes can be used.
  • Also the RN2483 can be sent to sleep in order to minimize power consumption. The goal is how to find out how to send this module to sleep and wake it up again, and what implications this has to the current connection to TTN.

Last but not least …

Wiring

I use a RN2483A/RN2903A breakout board that I ordered for about €30 at Tindie: https://www.tindie.com/products/DrAzzy/lorawan-rn2483rn2903-breakout-board-assembled/

RN2483 Breakout Board Pinout

Level Shifter Option: As the Espruino Pico already operates at 3.3V, there should be no need for a level shifter. However, if you like to play around with this module by simply attaching a (FTDI) Serial-to-UART cable which has 5v levels, then you probably want to have a level shifter and a voltage regulator at hand.

The Espruino Pico gets its power from a 9V battery. It has a built-in voltage regulator that converts the volate to 3.3V. Espruino Pico Pinout

The MaxBotix MB1242 consumes a consiberable amount energy, even in idle mode, so a Adafruit LM3671 3.3V Buck Converter Breakout is used as kind of “switch” to cut-off the power supply for this sensor when it is not in use. Yes, people say that cutting off the power supply is not sufficient to make sure that the MB1242 really no longer consumes any energy, but in my case this trick was very effective.

Wiring on Breadboard

Preparation in TTN Console

Source code: https://github.com/wklenk/espruino-pico-rn2483-ttn-basic

https://console.thethingsnetwork.org/ You first need to create an application in TTN console, and create a device within this application.

Select OTAA (Over the Air Activation) as Activation Mode

After creation of the device, copy and paste the values for Device EUIApplication EUI and App Key to the code:

// LoRaWAN Over the Air Activation (OTAA)
// Copy these values from the Device Overview of the TTN console.
const deviceOTAAConfiguration = {
  'deviceEUI': '0000000000000000',
  'applicationEUI': '0000000000000000',
  'appKey': '00000000000000000000000000000000'
};

Transfer code to Espruino Pico and run it

I won’t explain here how you work in general with Espruino products, you already should be familiar with that or make yourself familiar by other means than this project.

  • Connect the Espruino Pico via USB,
  • use the Espruino Web IDE to transfer the code (I still use the native version for Windows)
  • Run the code by entering onInit(); on the Pico

Note that the Pico won’t go into stop mode until connected to USB. You need to connect a battery (or another power source) to try that out.

https://www.espruino.com/Power+Consumption

Example output

 ____                 _
|  __|___ ___ ___ _ _|_|___ ___
|  __|_ -| . |  _| | | |   | . |
|____|___|  _|_| |___|_|_|_|___|
         |_| espruino.com
 2v06 (c) 2019 G.Williams
>
>onInit();
=undefined
["sys get ver\r\n"
] "R"
] "N2483 1.0.5 Oct"
] " 31 2018 15:06:5"
] "2\r\n"
Version: RN2483 1.0.5 Oct 31 2018 15:06:52
["mac reset 868\r\n"
] "o"
] "k\r\n"
["mac set adr on\r\n"
] "o"
] "k\r\n"
["mac set appeui 70B3D57EXXXXXXXX\r\n"
] "o"
] "k\r\n"
["mac set deveui 00022CBFXXXXXXXX\r\n"
] "o"
] "k\r\n"
["mac set appkey 625C109CA95ADA37E9D0EA64XXXXXXXX\r\n"
] "o"
] "k\r\n"
["mac join otaa\r\n"
] "o"
] "k\r\n"
] "a"
] "ccepted\r\n"
Result: accepted
RN2483 go to sleep
["sys sleep 86400000\r\n"
RN2483 sent to sleep
Periodic task started.
RN2483 wake up
] "\u0000"
] "ok\r\n"
["\r\n"
] "inv"
] "alid_param\r\n"
wake up finished
["mac tx uncnf 1 42\r\n"
] "o"
] "k\r\n"
] "m"
] "ac_tx_ok\r\n"
RN2483 go to sleep
["sys sleep 86400000\r\n"
Periodic task done.
> 

Results

  • The experiment up to now runs for more than 6 weeks now, longer than any battery powered circuit I did before. Update 2020-12-29: Died after 45 days, which is a little disappointing.
  • Maybe it could be even more power-efficient if we could eliminate the voltage regulator on the Espruino Pico and use an appropriate battery instead
  • The Microchip RN2483 module has quite helpful in keeping the complexity of the LoRaWAN protocol away from the project, especially suprised with its duty cycle enforcing mechanisms and adaptive data rate functionality. It also has a nice functionality to send it to sleep and wake it up when needed again. https://www.thethingsnetwork.org/docs/lorawan/duty-cycle.html

To visualize the measurements, I used

Running now for more then 6 weeks, hanging on the ceiling of my garage

Grafana Output - Running 6 weeks

Sending a distance value every 30 minutes (well, there are some few losses)

Grafana Output - Last hours

Some similar project with focus on energy consumption and also using an ultrasonic sensor can be found here:

https://www.hackster.io/Amedee/low-power-water-level-sensor-for-lorawan-the-things-network-96c877

Flat Rate for NB-IoT: Accessing the 1NCE Management API

Background

1NCE is an IoT connectivity provider with a disruptive pricing model:
The company offers a so-called “IoT Flat Rate”:
For 10 € you get a SIM card with 10 years lifetime and a flat rate traffic volume of 500 MB and 250 SMS.

As a maker of IoT devices communicating using NB-IoT you now question yourself:

  • Would such a flat rate be sufficient for my use-case?
  • What is the current traffic consumption of my device?
  • What effects does it have on the traffic consumption if I change the communication pattern in one or the other way?

Altough some RF modules like the Quectel BG96 offer counters to query the number of bytes sent and received, which already give a good indication about the traffic consumption, these perhaps don’t take into account the traffic on the lower levels of the communication stack, like UDP or TCP.

So why not directly ask 1NCE for the “consumed” traffic volume?

1NCE offers an Application Programming Interface that allows users to implement the most important functions of the Customer Portal into a customer’s environment. The supported functions of the 1NCE API generally enables the customer to manage, control and monitor the 1NCE SIM Cards from an external system.

https://1nce.com/en/help-center/tutorials-documentations/api-functionality/

For example, one can query for a list of SIM cards using a HTTP GET request to

https://api.1nce.com/management-api/v1/sims

… or get information about the left over flat rate traffic volume with a HTTP GET request to

https://api.1nce.com/management-api/v1/sims/my-iccid/quota/data

There is a Swagger UI interface at https://api.1nce.com/management-api/swagger-ui.html that lets you play around with the functionality of the Management API.

This example application connects to this Management API, gets the SIM cards and
creates a Round-Robin-Database (RRDB) out of that. Then, the application queries the current traffic consumptions of the cards and enters it as sample into the RRDB.

And, as the final goodie, if you navigate to port 8080 with your web browser, a nice graph with the consumption rate of the last 24 hours is displayed.

flatrate-data-consumption

How to configure

Find the source code on github at https://github.com/wklenk/1nce-management-api-access-and-rrd4j-graphs

Put your client-id and client-credentials in configuration file application.yml:

ok-http-client:
# proxy:
# host: localhost
# port: 3128

1nce:
client-id: your-client-id
client-secret: your-client-secret
token-endpoint: https://portal.1nce.com/management-api/oauth/token

How to build

You need maven and a Java JDK to compile:

$ mvn clean package

How to run

$ java -jar target/1nce-management-api-access-and-rrd4j-graphs-0.0.1-SNAPSHOT.jar

On the first start, you may see something like this:

...
Tomcat started on port(s): 8080 (http) with context path ''
route=Route{/a.b.c.d:xxxx} response=Response{protocol=http/1.1, code=401, message=, url=https://portal.1nce.com/management-api/v1/sims?page=1&pageSize=100}
Request did NOT use an Authorization header.
Requesting a new Access Token by client credentials
Token endpoint returned status 200 OK and tokens OnceAccessTokenService.Tokens(accessToken=206bed3b-11e6-4253-97c4-cb410f6f08ab, tokenType=bearer, expiresIn=3560, scope=all, appToken=eyJhbGc...)
QuotaDataResponse(volume=488.29297, totalVolume=500.0, expiryDate=2028-11-20 00:00:00, lastStatusChangeDate=2018-08-20 11:25:03)
QuotaDataResponse(volume=499.98907, totalVolume=500.0, expiryDate=2028-11-20 00:00:00, lastStatusChangeDate=2018-08-20 11:25:03)
sample=update "./data-traffic-consumption.rrd" 1573889068:1.2275710689279974E7:11460.935679972172

If a RRDB file with name data-traffic-consumption.rrd already exist in the current directory, then it won’t be overriden.
If you explicitly want to start with a new RRDB file, start the application with option –override.

Fun fact: To meet the goal of 500 MB in 10 years, the traffic consumtion rate needs not to exceed the red line at 1.66 bytes/second. So, there is still much to optimize in the communication patterns of this example device.

 

Quectel BC95-G: How to update firmware

Playing around with the built-in CoAP protocol stack in the Quectel BC95-G module I found out that I need to update the firmware, because it was not possible to me to include the payload in a CoAP request, which is a rather crucial feature 🙂

Development board

I purchased a NB-IOT Module BC95 NBIOT Development Board Small System Board NB101 at aliexpress.com for about 17 Euros. Please note that you want to purchase a development board with a BC95-G on it, because only the “-G” product can be used with more than one NB-IoT RF band.

NB101-front-sideNB101-back-side

I use a FTDI cable to connect the development board to my PC via USB. Having the appropriate FTDI drivers installed, there is an additional COM port. I use the application PuTTY to open a serial connection to this COM port at 9600 Baud.

Firmware revision

Use the AT command ATI to find out the firmware revision:

ATI

Quectel
BC95-G
Revision:BC95GJBR01A04

After the firmware update:

Quectel
BC95-G
Revision:BC95GJBR01A07

Firmware update

For updating the firmware, you need two things:

  • The Quectel Upgrade Tool QFlash (I used version 4.13)
  • The firmware update package (I used a package named BC95GJBR01A07.fwpkg)

The Quectel support pages will help you to get these things.

qflash-bc95g

Unfortunately, things get a little tricky here. The reason may be this particular development board I am using. May be other development boards behave in a different way:

  • Don’t use a baud rate of 9600 (like if you want to send AT commands), but instead use a baud rate of 115200, where the boot loader is listening for your commands when booting up.
  • When the QFlash application says “RESET” at the beginning of the update progress, you need to manually reset the BC95-G module. I simply turned off the power supply of the development board by plugging off the black Ground cable and then boot up the module by plugging it in again.

The QFlash application writes a log file log.txt in subdirectory Release\NB-IoT\1 that can help you to analyse any issues.

And now – have fun with your updated firmware!

Espruino on RAK8212: Digital Twin in Bosch IoT Suite

A few weeks ago, I wrote a blog post how to connect a Espruino powered RAK8212 device with AWS IoT. In this posting, I would like to showcase how to connect a RAK8212 device to the Bosch IoT Suite Platform using the Quectel BG96’s built-in MQTT stack.

The example will be a simple data logger, periodically sending sensor values using NB-IoT connectivity.

You can find the source code and other material on the following github repository:
https://github.com/wklenk/rak8212-espruino-bosch-iot-platform

The RAK8212 has quite a few sensors on board:

  • Temperature
  • Humidity
  • Barometeric Pressure
  • Ambient Light
  • Accelerometer
  • Magnetometer
  • Location (GNSS)

Assumptions

You are already familiar with the RAK 8212 and how to program it using Espruino.

Preparations

This example uses the Asset Communication package of the Bosch IoT Suite. There are quite a few things to configure before you can start, but there is an excellent tutorial that describes the steps in detail:

https://www.bosch-iot-suite.com/tutorials/connecting-a-simple-device-to-the-bosch-iot-suite/#book-and-configure

What you need to do (steps described in the tutorial):

  • Create a Bosch IoT Suite account
  • Subscribe to the Bosch IoT Suite Asset Communications package
  • Configure a namespace for your things (digital twins)
  • Create and configure a technical OAuth2.0 client
  • Generate a test token to access the APIs of the Bosch IoT Suite

About Vorto Information Models

The IoT Suite gives you full freedom how to model the Digital Twin of your physical devices. However, there is an open source project named “Eclipse Vorto” (https://vorto.eclipse.org) that allows you to compose an Information Model of your device from several so-called Function Blocks. So, like with Lego Bricks, you can compose your device from other building blocks that can be shared by different devices.

Actually, to explain the full functionality behind that would go to far here …

The Information Model for the RAK8212 looks as follows:

vortolang 1.0
namespace org.klenk.connectivity.iot
version 1.0.0
displayname "RAK8212"
description "Information Model for RAK8212"

using com.bosch.iot.suite.example.octopussuiteedition.Humidity; 1.1.0
using org.eclipse.vorto.tutorial.Temperature; 1.0.0
using org.eclipse.vorto.Illuminance; 1.0.0
using com.bosch.iot.suite.example.octopussuiteedition.Magnetometer; 1.1.0
using org.eclipse.vorto.BarometricPressure; 1.0.0
using com.bosch.iot.suite.example.octopussuiteedition.Accelerometer; 1.1.0
using org.eclipse.vorto.Location; 1.0.0
	
infomodel RAK8212 {

	functionblocks {
		temperature as Temperature
		barometricPressure as BarometricPressure
		humidity as Humidity
		magnetometer as Magnetometer
		accelerometer as Accelerometer
		illuminance as Illuminance
		location as Location
	}
} 

One of the several advantages to model a device in an Information Model is that you now can generate code or other helpful things from it, like for instance the device provisioning script.

Again, this is out of scope for now, but the provisioning script generated from this Information Model is available in this repository (rak8212-device-provisioning-msg.json)

Have a look at this Youtube channel of Tim Grossmann for more inspiration about Eclipse Vorto. https://www.youtube.com/channel/UC9_Bk9247GgJ3k9O7yxctFg/featured

Device Provisioning

Again, the Bosch tutorial at https://www.bosch-iot-suite.com/tutorials/connecting-a-simple-device-to-the-bosch-iot-suite/#book-and-configure gives detailed instructions how to register your device with the Bosch IoT Suite.

But, as the RAK8212 has its own Information Model and provisioning script, use this one that is available in this repository:

rak8212-device-provisioning-msg.json

{
  "id": "<your namespace>:rak8212",
  "hub": {
    "device": {
      "enabled": true
    },
    "credentials": {
      "type": "hashed-password",
      "secrets": [
        {
          "password": "<your-device-password>"
        }
      ]
    }
  },
  "things": {
    "thing": {
      "attributes": {
        "thingName": "RAK8212",
        "definition": "org.klenk.connectivity.iot:RAK8212:1.0.0"
      },
      "features": {
        "temperature": {
          "definition": [
            "org.eclipse.vorto.tutorial:Temperature:1.0.0"
          ],
          "properties": {
            "status": {
              "value": 0.0,
              "unit": ""
            }
          }
        },

How to send telemetry data

To send telemetry data and in this way update the Digital Twin, you need to built up a Eclise Ditto message and publish it to topic telemetry.

sendAtCommandAndWaitForPrompt('AT+QMTPUB=0,1,1,0,'
    + JSON.stringify("telemetry"),
    5000,
    '{' +
    '  "topic": "org.klenk.connectivity.iot/rak8212/things/twin/commands/modify",' +
    '  "headers": {},' +
    '  "path": "/features/temperature/properties",' +
    '  "value": {' +
    '    "status": {' +
    '      "value": ' + currentTemperature + ',' +
    '      "unit": "Degree Celsius"' +
    '    }' +
    '  }' +
    '}',
    '+QMTPUB:'
  )
     .then( ... )

Using the Quectal BG96 module’s command AT+QMTPUB to publish a message to a MQTT topic, this code section updates the feature “temperature” of the Digital Twin.

How to subscribe to “modified” events of the Digital Twin

The built-in MQTT client of the Quectel BG96 makes it easy to subscribe to topics: You just need to use the AT+QMTSUB command to subscribe to topic command/+/+/req/#.

sendAtCommand('AT+QMTSUB=0,1,' + JSON.stringify("command/+/+/req/#") + ',1', 15000, '+QMTSUB:')
  .then( ... )

Now, whenever a message is received on this topic, the MQTT client creates an output line starting with +QMTRECV: , like the following one:

] "\r\n+QMTRECV: 0,3,\"command///req/2240e49bdde-57ad-4652-b557-70f2dcf7"
] "41c0replies/modified\",\"{\"topic\":\"org.klenk.connectivity.iot"
] "/rak8212/things/twin/events/modified\",\"headers\":{\"sec-fetch-mode\":\"cors\", 
....

You can see this output if you put the debug mode on. In case someone changed the device’s digital twin, we will receive a Eclise Ditto “modified” message like the following one:

{
  "topic": "org.klenk.connectivity.iot/rak8212/things/twin/events/modified",
  "headers": {
    "sec-fetch-mode": "cors",
    "referer": "https://apidocs.bosch-iot-suite.com/?urls.primaryName=Bosch%20IoT%20Things%20-%20API%20v2",
    "sec-fetch-site": "same-site",
    "accept-language": "de-DE, de;q=0.9, en-US;q=0.8, en;q=0.7",
    "correlation-id": "0e49bdde-57ad-4652-b557-70f2dcf741c0",
    "dnt": "1",
    "source": "iot-suite:service:iot-things-eu-1:50058525-a6ed-4401-9984-f678cd509323_things/full-access",
    "version": 2,
    "accept": "application/json",
    "host": "things.eu-1.bosch-iot-suite.com",
    "content-type": "application/vnd.eclipse.ditto+json",
    "accept-encoding": "gzip, deflate, br"
  },
  "path": "/features/temperature/properties",
  "value": {
    "status": {
      "value": 21.61,
      "unit": "Degree Celsius"
    }
  },
  "revision": 525,
  "timestamp": "2019-09-22T10:58:49.666Z"
}

This one says that the feature “temperature” was modified.

In order to process this message, you just have to write a “handler” that deals with this incoming message.

Conclusion

As the Quectel BG96 module already has built in a MQTT stack, it is simple to communicate with the Bosch IoT Suite using this protocol. Using Espruino, one can efficiently build IoT prototypes without a big learning curve in regards to the programming of embedded devices.

However, it needs to be evaluated if MQTT in combination with this kind of verbose Eclise Ditto messages is actually a viable way for NB-IoT communication, where every transmitted byte counts to save energy and communication costs.

NB-IoT Explorer

Explore Narrowband IoT (NB-IoT) Radio Networks in a comfortable, user-friendly way.

Animated picture

In this project, two extremely powerful components come together:

The Espruino Pixl.js with its built-in JavaScript interpreter is easy to program. The (native) Espruino Web IDE lets you transfer the code to the device using Bluetooth LE, so there isn’t even a USB cable needed to connect the computer with the device. The Espruino Pixl.js device has a large LCD display with 128×64 pixels, and four buttons on each corner of the display allow highly interactive applications.

The Quectel BG96 module module on the NB-IoT Shield provides multi-band support, so it can be used in many different radio networks all around the world. The NB-IoT Shield provides direct access to the serial interface of the BG96 module, so you also can try things out using a FTDI cable, which can be useful if you want to do firmware updates. The BG96 module provides a wide range of AT commands for a lot of purposes, and even provides high-level protocol implementations like HTTPS and MQTT, which can be very useful for IoT applications. Additionally, it provides geo-positioning by its embedded GNSS chip.

Material

  • Espruino Pixl.js – Bluetooth microcontroller programmable in JavaScript with built-in LCD display and Arduino footprint.
  • Dragino Nb-IoT Shield QG96 – Arduino shield that hosts a Quectel BG96 module. This module supports multiple bands for NB-IoT. Except NB-IoT, It also support LTE Cat M1 and EGPRS.
  • A SIM card from a network provider supporting NB-IoT.
  • Micro USB cable for power supply
  • Optional: GNSS Active Patch Antenna
  • Bluetooth Low Energy Dongle for your computer, if it not already supports BLE natively.

Preparation

  • Get yourself familiar how to connect the (native) Espruino Web IDE with the Espruino Pixel.js board via Bluetooth LE.
  • Play around with the code examples for the Pixl.js.
  • The 5v pin on the on the Pixl.js will be used to power the NB-IoT shield. There is a solder jumper near the LCD connector labelled “3.3 5V Vin”. Short Vin to 5v – the 5v pin will be connected to 5v (when connected via USB) or whatever the voltage provided on Vin is. Read more about shield power at https://www.espruino.com/Pixl.js#shield-power. (Using the 3.3v at the output of the power regulator does not provide enough power when the BG96 starts to transmit).
  • If you are good in soldering, you can consider to physically remove the big square socket connector at the reverse side of the NB-IoT shield, because it is in the way when the Pixl.js and NB-IoT shield are connected.
  • Connect the Pixl.js with the NB-IoT shield by its Arduino connectors.
  • Optionally: Attach the GNSS Patch Antenna.
  • Get yourself a SIM card suitable for NB-IoT and insert it into the SIM card holder of the NB-IoT shield. Note: The cut corner of the SIM card has to point in your direction when inserting it. If you don’t take attention, then there is a chance that you insert the SIM card in the wrong orientation.
  • Load the JavaScript code from https://github.com/wklenk/nb-iot-explorer into the (native) Espruino Web IDE.
  • Check for the JSON structure named connection_options and provide the appropriate settings for the NB-IoT network provider of your choice.var connection_options = { band: “B8”, apn: “iot.1nce.net”, operator: “26201” };
  • Turn on Minification in Settings -> Minification. Choose Closure (online) – Simple optimizations.
  • Send the application to the Pixl.js.
  • If you want to keep the application in the Pixel.js even after re-booting the Pixl.js, then save it to flash by typingsave();
     ____                 _
    |  __|___ ___ ___ _ _|_|___ ___
    |  __|_ -| . |  _| | | |   | . |
    |____|___|  _|_| |___|_|_|_|___|
             |_| espruino.com
     2v01 (c) 2018 G.Williams
    
    save();
    =undefined
    Compacting Flash...
    Calculating Size...
    Writing..
    Compressed 40000 bytes to 20601
    Running onInit()...
    Press RESET button on NB-IoT shield if onInit() was called interactively.
  • To connect to the NB-IoT network, press the RESET button on the NB-IoT shield now. This is not necessary when you boot the device. In this case, the NB-IoT shield will reset by its own. When the application detects that the BG96 module is ready for operation, it will flicker the background light of the LCD display 5 times.
  • Now it is time to wait, as it may take seconds to a few minutes now for the BG96 module to manually register at the NB-IoT radio network. The activity LED should flash in the rhythm “on-off-off-off” periodically to indicate network search.
  • Once the BG96 module is connected, the splash screen on the LCD display will disappear, and other screens will appear. Use the two buttons on the corners of the right side of the display to cycle up/down trough these screens.

Screens

The button on the left top corner can be used to toggle the backlight of the LCD display. Use the two buttons on the corners of the right side of the display to cycle up/down trough the following screens.

When taking the screenshots, a NB-IoT SIM card of provider 1nce.com was used.

Screen: Registered Network

Registered Network

Displays the name of the registered network. The “Registered Public Land Mobile Network” (RPLMN) is identified by a globally unique PLMN code, which consists of a MCC (Mobile Country Code) and MNC (Mobile Network Code). The screenshot shows MCC 262 for Germany and MNC 01 for Deutsche Telekom.

Screen: Registration Status

Registration Status

Displays the Network Registration Status and the received signal strength (RSSI). On successful connection to the radio network, the status should be Registered Home Network or Registered Roaming.

Screen: Cell Information

Cell Information

Displays the two-byte tracking area code (TAC) in hexadecimal format, the 3 1/2 byte (28 bit) E-UTRAN cell ID in hexadecimal format, the eNB ID in decimal format (E-UTRAN cell id without 8 bit sector information) and the sector of the base station antenna.

This information can be used to look up the position of the base station tower in a map, with a service like https://www.cellmapper.net:

  • Enter Provider. The input field behaves a little bit strange. If RPLMN for example is “26201”, then you may need to enter “2621”.
  • Enter Network “4G – LTE”
  • In Input Area “Tower Search”, enter eNB ID and press Return

Cellmapper.net

Screen: Network Information

Cell Information

Displays network information such as the access technology selected, the selected band and the Channel ID.

Screen: IP Address

IP Address

Displays the IP address assigned to the BG96 module in the address space applicable to the PDP.

Screen: Geo Position

Geo Position

Displays the current geo position including Longitude, Latitude and Elevation, and the number of satellites received.

Screen: Date and Time

Date and Time

Displays the current date and time in UTC.