Coding fun – let me take that back

Effectively our bot is done.  It’s doing all the things we set out, receiving messages, searching for images, and replying when it finds one.

However, people are fickle things, and sometimes might get offended or otherwise want something the bot has published removed.

Admins of a chat group can remove anything anytime they want, but it’d be nice to tell the bot to remove it’s own messages.  Sounds easy, right?

It is and it isn’t.  We have two methods to accomplish this in Spark.  First, we could use the API to query for the last few messages, deleting them in reverse order.  Or we could use a data structure to remember the last few messages (10 in my case) and delete them as the user requests.

The power of programming, choice.  To that end, the act of deleting a message is easy, here’s the routine for it (within our spark module).

var deleteMessage = (msgId, callback) => {
 var sparkAPI = 'https://api.ciscospark.com/v1/messages/';

 var options = {
  url: sparkAPI + msgId,
  headers: {
   'Authorization': 'Bearer '
  }
 }

 request.del(options, (err, res, body) => {
  if(!err) {
   callback(res.statusCode);
  }
 });
};

At this point most of the functionality should be familiar to you.  We first define a call named deleteMessage, passing in a msgId(the actual message you want to delete) and a callback to deal with any output.

We define options, including the URL.  This time however we actually append the msgId to the URL.  Why? Because that’s what the API tells us to do.

The only new functionality is the API call:

request.del(options, (err, res, body) => {

We’ve only used .post and .get up to now, why .del?  Reading the documentation for the API it requires the delete method, and thankfully the request module has one.  It’s effectively a get by another name, just so the Spark back-end code can be clear that this is a deletion.

The only reply we expect back is an HTTP status code of 204, anything else and it didn’t work.

This is the easy part, the harder part is altering our main server code to record messages and then delete them in reverse order(newest first).  I’ve chosen to implement this with a double ended queue.  Other programmers will probably immediately have gravitated toward a stack or queue.  Here’s the notes about those

  • Stack
    • Push messages on “top”, remove them from the “top”
    • Upside, readily available in any language, generally easy to implement
    • Downside, you can typically only remove from the top.
      • I want to limit my message history to the newest 10 items.  To do this on a stack you’d have to rewrite it every time you hit 10, a performance nightmare.
      • There’s generally no way to limit how big the stack can get, and if you do it’ll just stop adding things, not functionality iwant
  • Queue
    • First in First out (FIFO) is the most common implementation
    • The First thing you add is the first thing you remove
      • Good for things that need to be done in order
      • Bad for us, we want to give the user the option to delete the Most Recent message, not the oldest

A double-ended-queue allows the best of both worlds.  It’s default implementation in this case is a stack, push all new messages on the top, and “pop” them off with the newest first.  But, it gives us a method to remove something at the bottom of the stack(the oldest message) effectively allowing us to keep the most 10 recent messages.  Programmers will also mention that these are also implemented with linked lists, but why do that complexity when you can simply use a module already defined, in this case double-ended-queue is an NPM module.

When I approach these types of problems, I write out pseudocode, this is always language independant, but helps visualize what I’m trying to do.  I’ve chosen to implement this with a

postMessage {
  success: 
   check queue size, if > 10:
    remove oldest item
   if < 10:
    add to queue
}
deleteMessage {
  check if queue is empty, if not:
   delete newest item
}

Logic done, let’s look at code.  Please note, all of this is done in the server module that we’re developing.  I’m ONLY going to review the code for the new queue and deletion.  I’ll post my final full server method at the end.

const dequeue = require('double-ended-queue');

var msgQueue = new dequeue();

 if(msgText.match(/^delete/i)) {
  if(!msgQueue.isEmpty()) {
   var deleteMsgId = msgQueue.pop();
   spark.deleteMessage(deleteMsgId.id, (statusCode) => {
  });
 }
 } else {
  spark.postMessage(link, messageDetail, (msgId) => {
   if(msgQueue.length >= 10) {
    msgQueue.shift();
   }

   msgQueue.push(msgId);
  });

This all happens after we’ve checked for a group message, and stripped out the Bot’s name (look at the last article for details on that). I’m only implementing this history function for group messages, if you get offended after private messaging the bot, just close the chat window.

I’m using regex to ensure that the first word is delete, in reality I should validate the second word too, but in this case if you say delete it just assumes to delete the last one.  An interesting excercise would be something like a ‘delete last X’ or ‘delete all’, but at this point if you’re following what’s happening the implementation of that should be fairly simple.

If it is a delete, we then check the messageQue, ensuring it’s not empty.  If it is nothing happens (maybe the script restarted, if it does all history is gone).

If the queue has entries, we pop off the first entry and pass it to deleteMessage.  At this point nothing is done with the statuscode.  If it succeeds you’ll see the message disappear in spark, if it fails nothing will happen.

If the first word isn’t delete, we assume a message needs to be posted.  I’ve stripped out the imgur search, but that occurs first and the return is send to postMessage.  The return callback from postMessage includes the JSON data of that exact message that it just sent. We are going to push this message on to the queue, but before we do that I check to ensure that it’s not full, if it is, I use the shift method to remove the oldest entry, and then push the newest into the queue.

And that completes this bot.  As I said, I’ll probably extend some of this functionality in a future post, and try to harden it a bit.

For reference, here is the complete server side code:

const express = require('express');
const bodyparser = require('body-parser');
const dequeue = require('double-ended-queue');

const spark = require('./spark.js');
const imgur = require('./imgur.js');

var botName = 'ImgurBot';
var msgQueue = new dequeue();

var app = express();
app.use(bodyparser.json());
app.use(bodyparser.urlencoded({ extended: true }));

app.all('/spark', (req, res) => {
 res.send();
 spark.getMessage(req.body.data.id, (messageDetail) => {
  if (messageDetail.personEmail !== botName + '@sparkbot.io') {
   if (messageDetail.roomType === 'group') {
   // Check if the bot is mentioned, if not ignore this
   if (messageDetail.text.split(' ')[0] === botName) {
    // Strip off the @ reference to the Bot
    msgText = messageDetail.text.split(' ').slice(1).join(' ');
     if(msgText.match(/^delete/i)) {
      if(!msgQueue.isEmpty()) {
       var deleteMsgId = msgQueue.pop();
       spark.deleteMessage(deleteMsgId.id, (statusCode) => {
       });
      }
     } else {
      imgur.search(msgText, (link) => {
       spark.postMessage(link, messageDetail, (msgId) => {
        if(msgQueue.length >= 10) {
         msgQueue.shift();
        }

        msgQueue.push(msgId);
       });
      });
     } // end of delete check
    } // end of botName check
   } else {
    imgur.search(messageDetail.text, (link) => {
     spark.postMessage(link, messageDetail, (msgId) => {
     });
    });
   } // end of group msg check
  } // end of bot own msg check
 });
});

app.listen(90);

Here is a link to a github repository with all the code

ImgurBot

Coding fun – message all the things

Today we actually make the bot interact, it will start to post messages to Spark.

We’ve setup nginx to listen, setup nodejs to actually listen and grab incoming spark messages.  We’ve setup a search to imgur, returning a text link, now we’re going to send it right back where it came from.

First I’m going to detail and explain the postMessage routine within the spark module:

var postMessage = (link, messageDetail, callback) => {
 var sparkAPI = 'https://api.ciscospark.com/v1/messages/';

 var postData = {
  files: [link]
 };

 if (messageDetail.roomType === 'group') {
  postData.roomId = messageDetail.roomId;
 } else {
  postData.toPersonId = messageDetail.personId;
 }

 var options = {
  url: sparkAPI,
  headers: {
   'Authorization': 'Bearer <YOUR TOKEN>'
  },
  body: postData,
  json: true
 };

 request.post(options, (err, res, body) => {
  if (!err) {
   callback(body);
  }
 });
};

Starting from the top

var postMessage = (link, messageDetail, callback) => {

Standard function definition, we’re passing in the string for the imgur link to an image, the details of the message that spark sent us(important for later), and the callback function we want to handle this with

 var sparkAPI = 'https://api.ciscospark.com/v1/messages/';

 var postData = {
  files: [link]
 };

Two variables, one to define the API of spark we’re working with.

The second variable will be the body of the document we send.  Post methods send any data inside the body of the document(with various encodings, see the link for details on that).  In our case we’re stating the files variable will be an array with our link.  We could send multiple files this way.  The array can also contain binary file data, but it’s convenient that we can pass it a url and it will fetch it within the spark client.

 if (messageDetail.roomType === 'group') {
  postData.roomId = messageDetail.roomId;
 } else {
  postData.toPersonId = messageDetail.personId;
 }

We passed in messageDetail so we could determine where the message came from.  If it came from a room, a group message, we need to reply back to that room.  Otherwise we reply back to the user that directly messaged us.

var options = {
 url: sparkAPI,
 headers: {
  'Authorization': 'Bearer <YOUR TOKEN>'
 },
 body: postData,
 json: true
};

Defining the options that will be passed to the request module.  This is just gathering all the data and flags we want to send.  Remember each authorization token is unique to spark’s bot, so fill yours in above.

I’ve also specified we’re sending pre-formatted json data, other languages (python) don’t work in native json so we’d need to pre-convert before using this.  NodeJS does, so happily it’s just set the flag and forget it.

request.post(options, (err, res, body) => {
 if (!err) {
  callback(body);
 }
});

Finally post the message to spark, returning whatever information the call gives us back (via a callback!).

Now let’s see it in context of our master server app

var botName = 'ImgurBot';

app.all('/spark', (req, res) => {
 res.send();
 spark.getMessage(req.body.data.id, (messageDetail) => {
  if (messageDetail.personEmail !== botName + '@sparkbot.io') {
   if (messageDetail.roomType === 'group') {
   // Check if the bot is mentioned, if not ignore this
   if (messageDetail.text.split(' ')[0] === botName) {
    // Strip off the @ reference to the Bot
    msgText = messageDetail.text.split(' ').slice(1).join(' ');
    imgur.search(msgText, (link) => {
     spark.postMessage(link, messageDetail, (msgId) => {
      console.log(msgId);
     });
    });
   } else {
    imgur.search(messageDetail.text, (link) => {
     spark.postMessage(link, messageDetail, (msgId) => {
      console.log(msgId);
     });
    });
   }
  }
 });
});

Most of this is self explanatory, there is one thing I’ll highlight below:

var botName = 'ImgurBot';

if (messageDetail.personEmail !== botName + '@sparkbot.io') {
 if (messageDetail.roomType === 'group') {
   // Check if the bot is mentioned, if not ignore this
   if (messageDetail.text.split(' ')[0] === botName) {
     // Strip off the @ reference to the Bot
     msgText = messageDetail.text.split(' ').slice(1).join(' ');

One of the spark restrictions is that Bots in group chats will only receive messages where they are mentioned.  This logic is checking for that.  First you have to define a variable with your bot’s name, make sure this matches including case.

We first check to ensure the message we just received, is not the message the bot just sent(that was a fun one to learn, recursion strikes again!).  Spark will append @sparkbot.io to any message with the bot, so in my case all it’s messages are from ImgurBot@sparkbot.io

Third we check if it’s a group message or not.  This is important because if it is our bot’s name will be the first word of the message.  We know this is true due to the requirement that bot’s be @mentioned within group messages(otherwise our script wouldn’t get any data in the first place).

If it is, we strip off the botName at the beginning (with a sanity check first) and create a new variable called msgText that contains everything else.

In the next post we’ll detail catching specific commands within msgText to allow the bot to do other things(like delete it’s own messages), but for now that’s the meat and potatoes that Imgur needs to search with.

We then search, post the message, and log the results.

At this point we are 4 callbacks deep:

app.all -> getMessage -> search -> postMessage

When you draw these out like above, you can really see how the power of a callback can write your program flow for you.

The power of a modular design like this, we can substitute anything we want for search(maybe doing weather data?  Maybe looking at an NMS for status on a device?).

One very common use for bots like this is in Network Operations, having your NMS alert you and send a message to spark.  In this way you aren’t worrying about email, and can potentially write scripts to interact via Chat (chatops) to acknowledge and potentially get more detail on alerts.

Coding Fun – So. Many. Images

For my fifth installment, we will use the Imgur API to search for random animated gifs, useful right?  Who says programming has to always be useful, distractions can be good for you.

Review the API documentation I linked above.  One thing you need to know about this API, you do have to register an account to use it, which has a quota.  Once you reach that quota I’d presume things stop working (I’ve never hit it).  You will need your API key to use this tutorial.

We’re going to create a new Imgur module, to handle all our Imgur logic which at this point is searching the gallery for images.

Here’s the framework for the module

const rp = require('request');

var IMGURAPIKEY = "<YOUR KEY>";

var search = (searchString, callback) => {
 var imgurl = 'https://api.imgur.com/3/gallery/search/';

 var options = {
  url: imgurl + 'viral?q_type=anigif&q_any=' + \
      encodeURIComponent(searchString),
  headers: {
   'Authorization': 'Client-ID ' + IMGURAPIKEY
  },
  json: true
 };

 rp.get(options, (error, response, data) => {
  if(!error && response.statusCode == 200) {
   for(var q in data.data) {
    if(!data.data[q].nsfw) {
     callback(data.data[q].link);
     break;
    }
   }
  }
 });
};

module.exports = {
  search
};

There isn’t much new here.  We load the request module, build our call to the API, and execute it.

One thing that probably warrants some discussion is the following

if(!error && response.statusCode == 200) {
 for(var q in data.data) {
  if(!data.data[q].nsfw) {
   callback(data.data[q].link);
   break;
  }
 }
}

Let’s break it down line by line

if(!error && response.statusCode == 200) {

In an HTTP request, two things can happen that we don’t want.

First would be an error, this is a true error, you couldn’t reach the server for some reason (DNS issue, routing problem, the server is offline, etc…).  If that happens the request module will set error to true, so we check to ensure it’s not (!) true.

Second would be that the server returns something other than HTTP status code 200.  Anything else would mean we got no data back, so there’s no point proceeding.  Ideally you want to add error handling and return something to the user.  My users know that when the bot never replies, it either had a failure or didn’t find anything.

for(var q in data.data) {
 ..
}

Imgur returns an array of data.  I obviously don’t want a chatbot flooding the room with data, so I’m going to loop over this array looking for the first image that meets qualifications.  This is setting up that loop.

if(!data.data[q].nsfw) {
 ..
}

I don’t want NSFW data, this is a work environment the bot is pasting to. Skip this if you don’t care about such things.  Notice that the variable q I set earlier is actually an array counter.  Python handles this differently which I might add as a bonus post, the differences in logic.  Probably should have used a better variable name than data earlier (since now we have the ultra confusing data.data) but I warned you that I’m not creative.

callback(data.data[q].link);
 break;

Remember my callback?  If you don’t know by now NodeJS uses these A LOT!  They are useful, and when you learn the recursion involved they start to make sense.  Here I’m sending the first link to an image through the callback.  I then break the for loop.  Again I don’t want to loop and flood my channel/users with a bunch of images, just one.

And that’s it, our main code can be adjusted to add this new search feature, here it is for your viewing pleasure.  I have added some logic to parse out the message content, nothing is new here so I won’t review it.

//Standard NPM modules
const express = require('express');
const bodyparser = require('body-parser');

const spark = require('./spark.js');
const imgur = require('./imgur.js');

var app = express();
app.use(bodyparser.json());
app.use(bodyparser.urlencoded({ extended: true }));

var botName = 'ImgurBot';

app.all('/spark', (req, res) => {
 res.send();
 spark.getMessage(req.body.data.id, (messageDetail) => {
  console.log('message:', messageDetail);
    if(messageDetail.roomType === 'group') {
      // Check if the bot is mentioned, if not ignore this
      if(messageDetail.text.split(' ')[0] === botName) {
        // Strip off the @ reference to the Bot
        imgurSearch = messageDetail.text.split(' ').slice(1).join(' ');
        imgur.search(imgurSearch, (link) => {
          console.log(link);
        });
      }
    } else {
      imgur.search(messageDetail.text, (link) => {
        console.log(link);
      });
    }
 });
});

app.listen(90);

One last thing to mention.  Within the imgur script I called this method

encodeURIComponent(searchString),

This is pretty critical.  I am encoding the search string to standard HTML characters, this should prevent any kind of exploit against the search.  Sanitizing things is important in web-security.

In reality this entire script needs to be hardened against attacks, again maybe in a future post I’ll do it and detail the changes.

Coding fun – API?

Fair warning, this is probably going to be a very long post. TLDR; APIs are useful.

Web-Hook is setup, NGINX is proxying and forwarding, NodeJS is listening, now we need to actually do something with the Spark data that is being received.

Remember the API?  Sure you do (here it is).  We will be using this extensively so pull it up in another window so you can reference it.

Let’s mock-up what we’re trying to accomplish.

  • Spark notifies us of a message
  • We grab the details of the message
  • We process those details

Easy right?  Wait, we have to grab details, why doesn’t spark just send us the details?  I’m not exactly sure, but likely so nothing super sensitive is being sent to a site that can potentially be hijacked.  So what are they sending us? It looks a little like this:

{ 
 id: 'garbldegoop',
 name: 'ImgurBot WebHook',
 targetUrl: 'https://example.com/spark',
 resource: 'messages',
 event: 'created',
 orgId: 'garbledegoop',
 createdBy: 'garbldegoop',
 appId: 'garbldegoop',
 ownedBy: 'creator',
 status: 'active',
 created: '2017-09-07T13:23:23.181Z',
 actorId: 'garbldegoop',
 data: 
 { 
  id: 'important garbldegoop',
  roomId: 'more important garbldegoop',
  roomType: 'direct',
  personId: 'garbldegoop',
  personEmail: 'email@email.com',
  created: '2017-09-08T15:24:52.948Z' 
 } 
}

Fill in a bunch of random strings where you see garbldegoop, each one is unique, and each is unique per message.  I’ve obfuscated mine so that some important details can be masked, but regardless we will be using code to break this big thing apart, and grab some details.

What fields do we need?  Depends what we’re going to do.  First we need to infer a few things from what we just received.

  • Where did this message come from?
    • Was it a Room/Space?
      • If it was, how can I tell where that is?
    • Was it an individual via a Direct Message?
      • What is their email address, or ID?
  • Where is the message itself?

Most APIs break all this stuff up so you can only grab the things you want, Spark is no different.  Using my example above, I know this message came in via a room.  How?

roomType: 'group'

Easy?  What’s it look like if it’s simply a message from a user?

roomType: 'direct'

Anyone can do this stuff, with that one field we can determine how our reply needs to happen.  But how do we get it?  Fun fact, Spark sent it to us directly.  Our code already handles it.  Remember app.all from the previous post?  No?  Here it is:

app.all(..) {
}

That doesn’t tell us much, let’s expand it and review what’s going on.

app.all('/spark', (req, res) => {
 res.send();
 console.log(req.body);
});

Simple.  Add this code to your app.all stanza and run it.  You can direct message your bot and it should dump the contents of the Spark web-hook to the console.  Welcome to debugging, printing things until you get what you want.  Buy what’s going on here?  Let’s detail it:

app.all('/spark', (req, res) => { 
  .. 
});

So, API calls use GET, or POST, or PUT, or DELETE, or UPDATE right?  That Express module we installed yesterday, it can handle them individually, or all at once.  Why would you do that?  It was designed to actually create APIs, we’re not doing that directly, instead we’re borrowing that functionality to get NodeJS listening to something.  By using the .all we are telling Node and Express to ignore what type of call comes in (through the reverse-proxy, remember that thing?)

When we registered the web-hook, we gave it a URI path (hint, /spark) so it’s happily passing it through.  We have to tell Express to listen for a URI (app.all(‘/spark’…)).  Sure I could have used ‘*’ here, but that would have opened this thing up to ANY web request making it through the proxy, I like security too, so lets just make sure things line up as much as possible.

Now, if you read the documentation for express, it’s actually expecting the next part to be a callback.  What is a callback?

Wouldn’t it be fun if we could just tell computers how to handle data on the fly?  We didn’t used to, programmers generally had to hard-code every possible way data could be handled, no flexibility in that.

A callback is a function that YOU write, telling some other function how to handle data.  Welcome to recursion(another fun, but very abstract CompSci topic).  This solves our future data problem.  All a programmer has to know is what data is being returned, and can act on it.

app.all('/spark', (req, res) => {
 ..
});

Everything in bold is a function, the callback.  Here’s how we could write it differently

app.all('/spark', callbackFunction);

For readability it might make sense to do this, but the more you get used to NodeJS the more you’re going to realize almost everything uses a callback.  It would get really tedious to write these out and substitute them everytime you wanted to implement a callback, right?  We are coders, we like to do things lazy and easy, so write the function in-place.

Speaking of, see the funny => thing in our callback?  That’s a really interesting way to create functions in NodeJS.  Remember, we like to be lazy.  The link I gave you is a very thorough example of why we use these, read it and research “arrow” functions.  This will help you understand some of why we use them (other than laziness!).

So, Spark is sending us some data, our app.all(..) is receiving that data, and we have some kind of callback function thing doing stuff.  What’s it doing?

 res.send();

Back to relating things to Network Engineers.  Ever typed a command wrong on a cisco IOS cli prompt?  What’s the first thing you always forget to disable, which locks the session up until DNS times out?  no ip domain-lookup right?

Spark is sending us information.  It’s using HTTP to send us that data.  HTTP expects to be answered, even if it’s a return of blank data.  We don’t want to be the bad dns-lookup.  Spark needs to be answered, so we do it.  We send back a reply, it’s blank, the equivalent of sending an HTTP status of 200 with nothing in it.  Spark is happy, we’re happy.

console.log(req.body);

Take whatever Spark sent us (req.body) and log it to the console.  I wish it were this simple, remember earlier we downloaded a new module (body-parser) and included it?

const bodyparser = require('body-parser');
..
app.use(bodyparser.json());
app.use(bodyparser.urlencoded({ extended: true }));

Spark is using a POST method to deliver our information.  POST embeds all it’s data in the body of an HTML document.  Sometimes this is plaintext, urlencoded, obfuscated, whatever.  Body-parser includes the magic needed to interpret all this stuff and get it into a readable format we can use, in our case Spark is sending us JSON so we’re processing it.

Go read the documentation for body-parser for more information, it’s really good at explaining this.

So req.body contains that formatted data, we dump it to screen.

I told you this would be long, go get a drink, stretch your legs, and come back.  I’ll wait.

 

—–

 

Back?  Good, let’s start working with the spark API itself, it gave us information, it’s only fair we USE it for something.

We’re working with JSON.  Probably pretty good we’re using JavaScript to process it, being that it was created for that language originally.  JSON is really easy to process within NodeJS, you assign a variable some java script and magic happens in the background.

We’re going to rewrite app.all now:

app.all('/spark', (req, res) => {
  res.send();
  var sparkDetail = req.body;
  var messageId = sparkDetail.id;
  spark.getMessage(messageID, (messageContents) => {
   console.log(messageContents);
  });
);

As a refresher, here’s the JSON detail that spark sends us

{ 
 id: 'garbldegoop',
 name: 'ImgurBot WebHook',
 targetUrl: 'https://example.com/spark',
 resource: 'messages',
 ...
}

Breaking down our changes to app.all

var sparkDetail = req.body;

Creating a new variable, named sparkDetail.  It contains the entire JSON that spark sends us.

var messageId = sparkDetail.id;

I told you earlier, when you assign JSON to a variable in NodeJS it just magically knows how to handle it.  We did that with the body, Spark only sends JSON in the body.  JSON is simply a series of variables and values being sent to us.  We need to extract data from those variables.  Specifically we need the variable called ID, and the contents of it.  It’s the very first thing Spark sends us in the JSON document.  Since sparkDetail contains the entire JSON data, I can simply request the ID variable by appending .id to it.

If I wanted to get the name, it’d be sparkDetail.name, the targetUrl would be sparkDetail.targetUrl, and so on…

spark.getMessage(messageId, <<CALLBACK>>);

I ommited the callback for clarity, we are calling a new function here (that doesn’t exist yet!) in the module spark(we haven’t created it), the function is getMessage.  It expects the ID of the message we want to get, and will return some data.  That data will be handled by a callback.

I’ll rewrite it with the callback

spark.getMessage(messageID, (messageContents) => {
 console.log(messageContents);
});

Go lookup arrow functions if you didn’t earlier, understand them before trying to parse this out.  The callback function written out explicitly is:

function (messageContents) {
  console.log(messageContents);
}

I know, I know, arrow functions aren’t full functions.  Go ahead and comment and tell me that, otherwise get over it and move on if you understand that concept.  The getMessage function will return data for us that gets put into the variable called messageContents.  We then log it to console.

Did you catch what’s wrong?  The spark module isn’t loaded, in fact we haven’t created it yet.  Let’s do that now.  Edit a new file and name it spark.js, here’s the barebones contents:

const request = require('request')

var getMessage = (msgId, callback) => {
  // get spark msgId
};

var postMessage = (body, callback) => {
  // post spark message
};

var deleteMessage=(msgId)=>{
  //
};

module.exports = {
  getMessage,
  postMessage,
  deleteMessage
};

Really only one new thing to learn in this, what does module.exports do?  Well, when you create a modular program you need to let your main program know what functions or variables you are giving it access to.  In our case we’re writing all the logic to interact with the Spark API directly in this module, so we need to expose the functions we want the consumer to use.  module.exports does that for us.

Back to the Spark API, this time we are looking at messages.  Specifically we are going to use the GET method specifying the messageId, let’s rewrite our getMessage function and then go through some of the highlights

var getMessage = (msgId, callback) => {
  var sparkAPI = 'https://api.ciscospark.com/v1/messages/';

  var options = {
    url: sparkAPI + msgId,
    headers: {
      'Authorization': 'Bearer <YOUR BOT KEY>';
    },
    json: true
  };

  request.get(options, (error, response, data) => {
    if(!error && response.statusCode == 200) {
      callback(data);
    }
  };
};

First we define a couple of variables, the first is obvious, the path to the Spark API method we want to call.  The second however needs a little discussion.

RESTful APIs generally require some sort of authentication.  Some use OATH, some don’t.  All of them generally expect some kind of token to be presented, to prove you are who you are, or to track your usage (billing!).  Spark is no exception, the only difference is that it generates your bearer token when you create your bot.  You did write that down right?  No, well good thing you can use the Test mode of the API to get it.  When you put it into Test mode you should see the Bearer <KEY> line show up, just copy it and substitute it where appropriate.

Let’s run this code and see what our output is.  Oh, we need to update our server code to call it and register the callback.

const express = require('express');
const bodyparser = require('body-parser');

const spark = require('./spark.js');

var app = express();
app.use(bodyparser.json());
app.use(bodyparser.urlencoded({ extended: true }));

app.all('/spark', (req, res) => {
 res.send();
 //console.log("**", req.body);
 spark.getMessage(req.body.data.id, (messageDetail) => {
    console.log(messageDetail);
 });
});

app.listen(90);

Add this to your server file, and run it.  You should be able to Direct Message your bot’s name, and the output on the screen will be the details of that message.

That will end this one, next we’ll work on using the Message data to search Imgur.

Coding fun – listening yet?

We have a web-hook configured, NGINX is up and reverse-proxying the data, now we work on the actual NodeJS script that’s listening for data, you know the actual work.

As usual, I’m not going into specifics on how to get a NodeJS environment up, there are far too many tutorials that are good at this already.  This post will detail the script specifics that I used to get things running.

That out of the way, assuming you have an environment running, we need to figure out a way to get NodeJS to listen for that incoming web-hook and process it.  NodeJS has the NPM system of module installation, which has so many different modules to use we won’t be hard pressed to find something.

Enter the module Express, we need to install it.  While we’re at it we’ll install another module called body-parser, more on that later.

npm install express --save
npm install body-parser --save

This will install the module, and add it to the package.json file for future install (in case you want to get this running in Docker, hint I will be doing this at the end).

Now we can start our barebones script to listen for those incoming web-hooks

const express = require('express')
const bodyparser = require('body-parser')
var app = express();
app.use(bodyparser.json());
app.use(bodyparser.urlencoded({ extended: true }));
app.all('/spark', (req, res) => {
  res.send();
  //Parse all the spark/imgur stuff in here...
});

app.listen(90);

I’ll go through this in detail now.

const express = require('express')
const bodyparser = require('body-parser')

Remember those modules we installed?  Here we’re including them, essentially giving us more functionality.  I do this with constants (things that cannot be changed).  This helps me with readability, as I know most constants are modules or hard-coded values that I need later.

var app = express();
app.use(bodyparser.json());
app.use(bodyparser.urlencoded({ extended: true }));

Ever hear Object Oriented programming?  We’re not doing that here, but we are borrowing some of it’s concepts.  The line var app = express(); is doing just that, we’ve created a variable (var) named it app (warning, I’m not very creative), that is an instantiation of the express object.  This is a really fancy way of saying we are creating a copy of the express module, and getting all the functions, constants, and variables it brings to the table..

There’s more nuances to Object Oriented programming instantiation, but it’s far too much to explain here.  Regardless, this is doing most of the work for us, getting the app ready to listen for those incoming web-hooks.

app.all(..) => {

}

I will go into much more detail on this in my next post.  The app object has a bunch of functions to setup and listen for web requests(like apache, or nginx the webserver).  It can handle GET calls, POST calls, PUT calls, everything you’d ever find in a modern REST API.  You didn’t know you were creating an API?  Surprise, you are (loosely).

app.listen(90);

And that’s all she wrote, this script once executed will listen on TCP port 90, and run until it crashes or we kill it.

Easy?  I think so.  Sure there are HUGE chunks of functionality missing, but at it’s core the act of getting a script to listen for incoming requests is done.

Next time, we fill out some of the meat of this thing.  I’ll explain what sort of data Spark is sending when messages come in (and how you can deduce that for yourself).  We’ll handle that data and grab the contents of a message.

Coding Fun – NGINX reverse proxy

YANRPA – Yet Another Nginx Reverse Proxy Article.  Seriously, there are a lot of good resources for reverse proxy.  I’m simply writing this as a documentation trail for what I did, so 6 months from now I’ll remember.

NGINX installation and setup will not be covered here, only the changes I’ve made to get the system to send data to my back-end server hosting the NodeJS script for the Imgur Spark Bot.

First, all of my config is done within nginx.conf, so go edit that with your editor of choice (VIM FTW!).  In my case, I’ve chosen to serve it up with a URI of /spark, seems simple and logical.  I won’t go into the intricacies of reverse proxy, just know that some back-end applications need to understand if their URI is changing.  If so they generally have a way to tell the application that.  Why is this important?  Because any links or paths need to be adjusted accordingly.  Remember that the reverse-proxy is simply acting like a client to the back-end server on behalf of the person asking IT for content.  It has to know where to go and what links to use.

Can you strip the path off?  Sure (with rewrite) but honestly it’s much harder.  In our case I’m going to ensure NodeJS knows that there’s a URI path that matches NGINX, and we won’t have these issues.

Here is my relevant nginx.conf stanza.

location /spark {
  proxy_pass http://10.0.1.51:90;
  proxy_redirect off;
  proxy_set_header Host $host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

Easy enough?  I’m telling the publicly exposed NGINX server to send any request for /spark to the server at 10.0.1.51 listening on port 90.  I’m adding some headers (in case I ever want to log where Spark requests come from, or more importantly to IP restrict them to known spark servers).

This is honestly all that’s required on the NGINX server, once it’s done we can proceed to finally creating some NodeJS code.

Employer Transitions

Inspired by Greg Ferro I have been considering, and subsequently made an employer transition recently.  This post will serve as some observations I’ve had after being in a new role for roughly 6 weeks.

  • It takes time to get comfortable with new surroundings
  • Things are different, but mostly similar everywhere in the Networking industry
  • Business and IT are frustrated at the state of our industry
  • ITIL is not fun

I’m also finding out that perspective is a funny thing.  I haven’t had many transitions in employer over the years, generally when I did it was a result of the business being purchased or closing, but it’s always been met with a measure of excitement and optimism.  Every time I’ve transitioned there are always great things, and not so great things.  This is the first time I’ve transitioned from larger to smaller, non-profit to for-profit, and yet the issues faced (political, outside of IT) are almost exactly the same.

If you haven’t read the article, go look for yourself and see if any of what Greg references sparks different perspectives.