Tag: ESP8266

A Database I Want to Use

If you’re only interested in connecting your ESP8266 to a database, skip to the bottom paragraph. If you’re not interested in databases, read the whole thing.

I am not interested in databases. As a software developer, I understand the value of a good database and I like using databases to store/retrieve/search/etc. data. The part I don’t like is the setting up, configuring, optimizing, scaling, maintaining, and other tasks required to keep a database up-and-running. I want to spend my time writing code rather than babysitting a database. I want to use a database, not run a database.

Nearly four years ago when playing with my then new ESP8266 I needed a database and I ended up going with REDIS and WEBDIS because it was the lowest bar to entry that I could find for my requirements. It has worked well-enough for my needs, but nothing to rave about. I’ve spent the past two years making something exponentially better.

In 2018 I joined Datastax which is the most important company behind the NoSQL database Apache Cassandra. Cassandra was not a database that I wanted to use. While the performance, scalability and reliability of Cassandra are unsurpassed, the complexity to get Cassandra ready to go and keep it running was a deal breaker for someone like me that isn’t interested in being a DB admin.

As a Senior Software Engineer on the Datastax Cloud Team I had various responsibilities, but my personal goal was to create a database I wanted to use. The team succeeded with Datastax Astra which is a “Cloud-native Database-as-a-Service built on Apache Cassandra”. That means I can just go to a website, fill in a few details, click a button, and seconds later I have a database I can use. It’s awesome.

My day job was working to make Datastax Astra a database I wanted to use and finally I am able to spend my weekends using it. I created an Arduino library called astra_esp8266 which makes it easy for an ESP8266 to communicate with a database. The source code is available on github but most users will probably just want to install the library into an Arduino IDE. After four years, with two of those years taking thousands of hours of my time, there is now a database that I want to use.

I love it when a plan comes together

After spending a lot of effort and encountering difficulties in creating pieces, I am often pleasantly surprised when the pieces come together quickly and easily.  This was the case for my latest home improvement tech project.  In my home, it seems like some areas are warmer than others–I realized that some variance will exist, but I wanted to reduce the overall difference between upstairs and downstairs.

The first step was to be able to measure the temperature or each area.  Thanks to my ESP8266 development boards, I am able to publish the upstairs temperature and publish it to a database and Bakboard.  With the new Nest thermostat and a little playing with the REST API, I was able to do something similar and publish the downstairs temperature to the BakBoard.  There are now four temperatures published on the Bakboard.

temps

I then wrote a simple Java program with that basically does the following:

  1. Get the temperature of the [Downstairs] thermostat
  2. Get the temperature of the [Upstairs] temperature sensor
  3. If the difference between the two temperatures is greater than 2 degrees, turn on the furnace fan

I had a little trouble figuring out how to turn on the fan, but this is the way I implemented it in Java:

public void runFan(String thermostatId, String authToken) throws Exception {
    final String rootUrl = "https://developer-api.nest.com";
    HttpPut httpPut = new HttpPut(String.format("%s/devices/thermostats/%s/fan_timer_active", rootUrl, thermostatId));

    StringEntity putEntity = new StringEntity("true");
    httpPut.setEntity(putEntity);
    httpPut.addHeader("Content-Type", "application/json");
    httpPut.addHeader("Authorization", "Bearer " + authToken);
        
    CloseableHttpClient httpclient = HttpClients.createDefault();
    try {
        CloseableHttpResponse response = httpclient.execute(httpPut);
            
        // We need to handle redirect
        if (response.getStatusLine().getStatusCode() == 307) {
            String newUrl = response.getHeaders("Location")[0].getValue();
            httpPut.setURI(new URI(newUrl));
            response = httpclient.execute(httpPut);
        }
           
        try {
            HttpEntity entity = response.getEntity();
            EntityUtils.consume(entity);
        } finally {
            response.close();
        }
    } finally {
        httpclient.close();
    }
}

Of course I want my code to run at regular intervals, but fortunately I had already figured out how to go about running a Java program every 15 minutes.  It was easy to toss everything into a Docker container and let it do its thing.

Here are a few notes/design decisions that I made when putting things together:

  • There are no changes to the basic functionality of the Nest thermostat.  It is not aware of the external temperature sensor and heats/cools as normal.  This means, even if something goes wrong in my code (or network connection or custom hardware or somewhere else), things can’t go too crazy.
  • My code does not control the length the fan runs–it starts the fan and lets the Nest take care of turning it off.  There is a default run time that can be set on the thermostat–in my case I set it to 15 minutes to match the run duration of my new program.
  • I have a two stage furnace and when just the fan is run it goes at half speed.  Even at full speed the furnace fan is pretty quiet, and at half speed we don’t even notice.
  • The thermostat only gives me the temperature in degree increments (if I were using Celsius it would be in half degree increments).  My homemade temperature sensor goes to greater precision, but it’s hard to say whether that greater precision provides better accuracy.  I went with a 2 degree variance threshold for enabling the fan to allow for rounding differences as well as accuracy differences between upstairs and downstairs temperatures.

As far as I can tell, everything came together smoothly and “just works” and has been for the past few weeks.  Occasionally I check the log to make sure it’s still running.  Once in awhile when I walk past the Nest I notice the fan icon indicating that the fan is running (and I can verify that by putting my hand near a vent).  The weather is still mild, so it will be interesting to see what happens when it gets colder (especially when I rev up the wood stove), but so far there seems less variance in temperature throughout the house.  I love it when a plan comes together . . .

My New Toy (Part 4)

I finally got to the point where my new toy is able to publish temperature information to a database, so next is to make that information available on BakBoard.  With the main pieces already deployed, the only thing necessary is to add the necessary connections.

First I wrote a some JavaScript to poll the Webdis/Redis server for the temperature.  For extra credit I also have it contacting the FAA Services REST API to get information about weather at the Portland airport.  Below is “temperature.js”.

function temperature() {
    setupTemperature();
    updateTemperature();
    window.setInterval(function(){ updateTemperature(); }, 100000);
}

function setupTemperature() {
    $('.temperature').html("<div class='pdx'/><div class='upstairs'/>");
}

function updateTemperature() {
    var myJson;
    var url = "http://192.168.1.35:7379/GET/temperature";
    
    myJson = $.getJSON(url, function(obj) {
        var f = parseFloat(obj.GET);
        var c = (f - 32) * 5 / 9;
        $('.upstairs').html("Upstairs: " + f.toFixed(1) + "F (" + c.toFixed(1) + " C)");
    });
    
    url = "http://services.faa.gov/airport/status/PDX";
    myJson = $.getJSON(url, function(obj) {
        $('.pdx').html("Airport: " + obj.weather.temp);
    });
};

And here’s a very simple HTML file that uses the script above to display the airport and upstairs (where I put my new toy) temperatures:

<html>
    <head>
        <!-- I downloaded jQuery from http://code.jquery.com/jquery-2.2.0.min.js-->
        <script src="jquery-2.2.0.min.js"></script>
        <script src="temperature.js"></script>
        <title>Temperature</title>
    </head>
    <body>
        <div class="temperature"></div> 
        <script>
            temperature();
        </script>
    </body>
</html>

Viewing that script in a browser showed:

Airport: 73.0 F (22.8 C)
Upstairs: 71.1F (21.7 C)

I basically added the bold HTML parts to the BakBoard .html file along with some css goodness and the temperature information is now displaying on BakBoard.

bakboardTemp

I had fun playing with my new toy and learned a lot.  The ESP8266 seems to have a lot of potential and I want to try more things.  Of course now that I have my only module in use, I may have to get another one for play . . .

My New Toy (Part 3)

In my previous “New Toy” post I managed to wirelessly publish a count, so I determined the next step to be to publish some “real” information: the temperature.

I’m not the first to use an ESP8266 module to publish temperature information and I found an excellent tutorial on Adafruit.  However, it was written for another (now discontinued) ESP8266 module with significant differences from my Electrow.  Also, the sample code provide has the module host a web server and allow clients to connect to it whereas I wanted my module to push the information to a database.  This meant I could use the tutorial as a guide, but there were still various things to figure out.

First I had to do the wiring to connect the DHT22 temperature and humidity sensor I purchased like the one used in the tutorial.  I ended up spending a lot of time and muttering various working words trying to get it to work right.  My module has onboard USB and a voltage regulator, so the wiring is much simpler, but I couldn’t get any readings.  I tried resistor changes, testing connections, and verifying voltages among other things.  Finally I discovered that when I specify pin 2 it means D04 (the fifth digital pin) on my board.  So here’s what I ended up with:

2016-08-02 15.35.48

  • The ground pin connects to the fourth pin of the DHT22 (black wire)
  • The D04 digital pin connects to the second pin of the DHT22 (white wire)
  • The 3.3v pin provides voltage (via the red wire) to the both the first pin of the DHT22 and it also goes through a 10K resister to the second pin.

 

 

The code is basically pasting together the Adafruit sample code with the code I wrote previously.  It looks like this:

#include <ESP8266WiFi.h>
#include <DHT.h>
#define DHTTYPE DHT22
#define DHTPIN  2

const char* ssid = "name of wifi network";
const char* pass = "wifi network password";
const char* ip = "192.168.1.35";
const int port = 7379;

// Initialize DHT sensor 
DHT dht(DHTPIN, DHTTYPE, 11); // 11 works fine for ESP8266
 
void setup(void)
{
  dht.begin();
  WiFi.begin(ssid, pass);
  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
  }
}
 
void loop(void)
{
    float temp_f = dht.readTemperature(true);

    WiFiClient client;
    client.connect(ip, port);
    String url = String("/SET/temperature/") + temp_f;
    client.print(String("GET ") + url + " HTTP/1.1\r\n" +
           "Host: " + String(ip) + "\r\n" + 
           "Connection: close\r\n" + 
           "Content-Length: 0\r\n" + 
           "\r\n");
    delay(5000);
}

I had already started up a Webdis/Redis server as described before, so I could verify it worked by running curl and confirming that it returned a value (in this case 72.68F).

$ curl http://192.168.1.35:7379/GET/temperature
{"GET":"72.68"}

Searching online, I found the ConnectSense CS-TH Wireless Temperature and Humidity Sensor that seems pretty much like what I made, but in a pretty box and costing $150 (granted it does come with some nice software as well).  Now that I have my own budget version, I can look to do something with that temperature data I’m collecting.

My New Toy (Part 2)

In the previous post, I implemented a simple counter with serial output and today I improved it.  The main reason I purchased the ESP8266 module in the first place was to get WiFi for cheap, so I wanted to try out the WiFi capabilities.  The resulting sketch is still a counter, but instead of publishing the count via the serial interface, it connects wirelessly to a database and publishes the count there.

To begin with, I needed a simple database that I could access via HTTP.  Redis is a simple key/value type database, but it doesn’t have an HTTP interface.  I found Webdis “A fast HTTP interface for Redis”.  To set things up quickly, I found that someone had already put everything together and published a Docker image on DockerHub.  So, through the magic of Docker, all I had to do to get Redis and Webdis up and running on my computer was run this magic command:

docker run -d -p 7379:7379 -e LOCAL_REDIS=true anapsix/webdis

I then wrote a sketch that would publish to my new database:

#include <ESP8266WiFi.h>
const char* ssid = "name of wifi network";
const char* pass = "wifi network password";
const char* ip = "192.168.1.35";
const int port = 7379;
int count = 0;

void setup() {
  WiFi.begin(ssid, pass);
  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
  }
}

void loop() {
  delay(5000);
  WiFiClient client;
  client.connect(ip, port);
  String url = String("/SET/count/") + count++;
  client.print(String("GET ") + url + " HTTP/1.1\r\n" +
               "Host: " + String(ip) + "\r\n" + 
               "Connection: close\r\n" + 
               "Content-Length: 0\r\n" + 
               "\r\n");
}

Basically, it connects to the wifi network (ssid) using the provided password (pass) and then publishes the count to the database (located at ip).  It publishes the new count value every five seconds.

To verify that it was working, I simply plugged this URL into my web browser :

192.168.1.35.7379/GET/count

It returns the current value of count:
getCount

So now I can not only program my new toy, but also use some of its wireless capabilities.  It’s not useful, but it is a good step to learning how to use the ESP8266.

My New Toy (Part 1)

Today my new toy arrived.  It’s an ESP8266 IOT WiFi Module.

ESP8622

Specifically I selected the “Elecrow ESP8266 IOT Board WiFi Module with Built in USB and Battery Charging” out of the many ESP8266 variants because of the following features:

  • Onboard USB (I find it easier than FTDI)
  • NOT breadboard friendly (Pins sticking up not down)
  • In stock and eligible for Prime shipping

To start with, I just wanted to verify that I could run some code on it.  Here’s what I did:

  1. Installed the Arduino IDE from https://www.arduino.cc/en/Main/Software.  I used the latest available (1.6.9).
  2. Configured the Arduino IDE to support ESP8266 boards:
    1. Opened up the preferences and added http://arduino.esp8266.com/versions/2.3.0/package_esp8266com_index.json to the “Additional Boards Manager URLs”.
    2. Opened the “Boards Manager”, found the “esp8266” listing, and clicked the “Install” button (using the latest 2.3.0 version).
    3. Since there wasn’t a specific Electrow entry, I selected “Generic ESP8266 Module” for the board type.
  3. Wrote some code.  Here is my very simple sketch to slowly count and send the number via serial: 
    int count = 0;
 void setup() {
 Serial.begin(9600);
 }
 void loop() {
 Serial.println(count++);
 delay(1000);
 }
  4. Ran the code.  I uploaded it to the module, opened the serial terminal, and saw that it was counting as designed.
    serialTerminal

Of course it took a bit of kicking and swearing to do that.  Here are a few of the things that I did before everything worked:

  1. Ran the Arduino IDE as root
  2. The upload speed is 115200, but the terminal speed is 9600 baud
  3. Change the reset method to “nodemcu”.
  4. Sometimes (but not always) hold the flash button and than hit the reset button before I could successfully upload my sketch.
  5. Switch USB cords (the first is a cheap, old cord that in recent years has only been used for charging devices).
  6. Check the port whenever I plugged in the module (it sometimes switched between /dev/ttyUSB0 and /dev/ttyUSB1 just to spite me).

There’s nothing exciting about my counting program, but by getting it running I confirmed that I can 1) Connect to the module 2) Upload code to it and 3) Run the code.  Now that I can do that, I can see what else I can make my new toy do . . .