Access Amazon RDS from AWS Lambda

Leave a reply

Recently I was working with serverless application created in NodeJs which was deployed to AWS Lambda. This app is connecting to MySQL database which was deployed to RDS. While I was testing in local machine it worked very well.

After deployed it stopped working. After struggling for half a day I was able to finally make it working. In this blog I am going to mention what it takes to make it working.

Different Security Group is the First Solution

This is really strange, if your Amazon RDS and AWS Lambda function shares the security group it will never connect. Not sure why it’s this way, but to solve the problem.

Create New Security Group and assign it to your Lambda function.
Create Other Security Group and assign it your RDS
Allow MySQL connection in RDS Security Group

To allow MySQL connection, add inbound rules. Select MYSQL/Aurora fromType and allow it from 0.0.0.0/0

Allow VPC Access to Lambda Function

Go to Roles page in IAM consle.
Create new role and add policy AWSLambdaVPCAccessExecutionRole
Assign this role to your Lambda function

Use Connection Pool in Your Code

While using serverless you have to be careful about asynchronous operations. Connection to MySQL is also asynchronous operations. If you do not handle it properly, lambda function will give you timeout as it’s expecting callback function to return result but because it’s async operations it will take time. So you have to explicitly let it know that, there is async operation so wait for it.

So for that create connection pool.

var mysql = require(‘mysql’);
var pool = mysql.createPool({
user : ‘username’,
password : “password”,
database: ‘database’,
host : ‘rds_end_point’
});

Now wait to get connection from the pool.

pool.getConnection((err, connection) => {
connection.query(‘SQL’,function(err, rows, fields) {
const response = {
statusCode: 200,
headers: {
“Access-Control-Allow-Origin” : “*” // Required for CORS support to work
},
body: JSON.stringify({
success: true,
rows: rows
})
};
connection.release();
callback(null, response);
});
})

As you can see in above code we are waiting for getting connection. Once we have the connection we query the database and get the result. After that we invoke the callback and release the connection so it can be used for next requests.

DialogFlow API – Use in Laravel

Leave a reply

DialogFlow is one of the best tool available for Machine Learning. It is developed and maintained by Google Team. You can use DialogFlow for various applications like Chat bot, Customer support service etc. In this blog I am going to explain how you can connect DialogFlow with Laravel using APIs.

DialogFlow

Generate Connection File

For using API, you first have to generate connection file which will be used for Authentication. For this here are the steps.

Login to DialogFlow Console
Select the Project
Go to Settings
Click on Service Account Link
You will be redirect to Google Project Console.
Select Service Account menu from left hand side.
Click on Add New Service Account
Follow the steps and add last step create and download JSON file.

This file will be used as connection credentials.

Integrate SDK in the Laravel App

First of all you have to download SDK using composer. Execute command in terminal

composer require google/cloud-dialogflow

This will download the SDK in vendor folder. Now set a link of credential file in ENV file.

GOOGLE_APPLICATION_CREDENTIALS= /path/to/your/file

Now you can use the API to connect to DialogFlow. Here are few examples.

First of all add following dependencies in your file to use the SDK.

use Google\Cloud\Dialogflow\V2\IntentsClient;
use Google\Cloud\Dialogflow\V2\Intent;
use Google\Cloud\Dialogflow\V2\Intent\TrainingPhrase\Part;
use Google\Cloud\Dialogflow\V2\Intent\TrainingPhrase;
use Google\Cloud\Dialogflow\V2\Intent\Message;
use Google\Cloud\Dialogflow\V2\Intent\Message\Text;

Get List of All Intents

Now use following code to get list of all the intents.

$intentsClient = new IntentsClient();
$parent = $intentsClient->projectAgentName(“YOUR DIALOGFLOW PROJECT NAME”);
$intents = $intentsClient->listIntents($parent,array(“intentView”=>1));
$allIntents = array();
$iterator = $intents->getPage()->getResponseObject()->getIntents()->getIterator();
while($iterator->valid()){
$intent = $iterator->current();
$allIntents[] = array(“id”=>$intent->getName(),”name”=>$intent->getDisplayName());
$iterator->next();
}
return Response::json(array(‘success’=>true,”allIntents”=>$allIntents));

Get Text Prediction

Use following code to get text prediction.

$sessionsClient = new SessionsClient($credentials);
$session = $sessionsClient->sessionName($projectName, uniqid());
$languageCode = ‘en’;
// create text input
$textInput = new TextInput();
$textInput->setText($text);
$textInput->setLanguageCode($languageCode);

// create query input
$queryInput = new QueryInput();
$queryInput->setText($textInput);

// get response and relevant info
$response = $sessionsClient->detectIntent($session, $queryInput);
$queryResult = $response->getQueryResult();
$queryText = $queryResult->getQueryText();
$intent = $queryResult->getIntent();
$displayName = $intent->getDisplayName();
$confidence = $queryResult->getIntentDetectionConfidence();
$fulfilmentText = $queryResult->getFulfillmentText();

Lumen Microservice Architecture

Leave a reply

I posted a blog on implementing Microservice Architecture. Here is the link. This blog is second part of it. In first post I just explained about Microservices and proposed architecture. In this post I will give practical example of Lumen Microservice.

Microservice Using Laravel

Lumen Microservice

You can either install Lumen in public folder of main Laravel app or you can have different installation all together.

composer global require “laravel/lumen-installer”

After it’s installed, now lets create an API.

$router->post(‘getDetails’,’APIController@getDetails’);

Now in a controller define the API.

public function getDetails(){

return response()->json(array(

“key1″=>”value1”,

“key2″=>”value2”,

“key3″=>”value3”,

“key4″=>”value4”,

“key5″=>”value5”

));

}

Now we can call these API either through our JavaScript app as Ajax request or we can call it in Laravel app using GuzzleHttp

Benefits of GuzzleHttp as mentioned on their website

Simple interface for building query strings, POST requests, streaming large uploads, streaming large downloads, using HTTP cookies, uploading JSON data, etc…
Can send both synchronous and asynchronous requests using the same interface.
Uses PSR-7 interfaces for requests, responses, and streams. This allows you to utilize other PSR-7 compatible libraries with Guzzle.
Abstracts away the underlying HTTP transport, allowing you to write environment and transport agnostic code; i.e., no hard dependency on cURL, PHP streams, sockets, or non-blocking event loops.
Middleware system allows you to augment and compose client behavior.

Install GuzzleHttp

Install it using Composer.

composer require guzzlehttp/guzzle

Once installed we can use in controller by name space.

use GuzzleHttp\Client;

Now call API using this client.

$client = new Client();

$response = $client->post(‘/lumenapi/public/getDetails’, [
‘form_params’ => []
]);

$data = json_decode($response->getBody()->getContents(), true);

Here in above code if you want to pass any param in POST request, you can do so inside form_params.

Above code is for the synchronous request. We can also have async request.

$request = new Request(‘GET’,’/lumenapi/public/getDetails’);

$client->sendAsync($request)->then(function ($response) {

$data = json_decode($response->getBody()->getContents(), true);

});

So this way we can build Lumen Microservice and consume it with Laravel app.

Laravel Cron Job Monitoring

Leave a reply

When we build applications using Laravel, we also have lots of Laravel cron jobs. Once the application is deployed on production server, it’s mandatory to monitor cron jobs. Many time it happens that due to certain issues crons are not executed. That may lead to lot of problems with your customers. As a server admin we can always login to server and check the logs. But what if we have a very simple page where we can see this logs.

Laravel Cron Jobs

Here in this blog I am going to explain a very simple approach to monitoring cron jobs. First of all for each cron jobs we will build a log file.

Build Laravel Cron Log File

We will use Laravel File class to create log files. First of all add to controller.

use File;

Add a variable to have counts of processed records with cron.

$count = 0;

Now inside foreach loop or any other loop you have increase this count.

foreach($records as $record){

$count++;

}

Once the cron execution logic is over build log file.

//Preparing log files on server.
$data = json_encode([“description”=>”Cron Execution Report”,”count”=>$count,”date”=>date(‘Y-m-d H:i:s’)]);
$fileName = ‘cronExecutionReport.json’;
File::put(public_path(‘/logs/’.$fileName),$data);

This way add logs files everywhere in your cron.

Read The Log File and Display It

Now that we have all the log files generated, we just have to read it and display it. Here is the logic for the same.

foreach (glob(public_path(“logs/*.json”)) as $file) {

$json = json_decode(file_get_contents($file), true);

echo ‘<span style=”font-weight:bold;font-size:25px”>’.$json[‘description’].'<br/>’;

print( ‘<span style=”font-weight:normal;font-size:20px”>Total Count : ‘.$json[‘count’].'<br/>’);

print( ‘<span style=”font-weight:normal;font-size:20px”>Last Processed Date : ‘.$json[‘date’].'<br/>’);

}
echo ‘<script type=”text/javascript”>setInterval(function(){window.location.reload()},10000)</script>’;

Add the above code in your controller function so that it will display following output and it will also refresh the page after 10 seconds to get the latest logs from server.

Cron Execution Report
Total Count : 10
Last Processed Date : 2018-11-09 10:56:01

So this way you can build a very simple Laravel Cron Job monitoring page.

Progressive Web App – Add Web Push Notification

Leave a reply

Welcome to Progressive Web App Series. If you have missed previous blog posts, please read from following links. In this blog post we will understand how to add Web Push Notification in PWA.

Why We Need Web Push Notification?

The best way to engage end users is through push notifications. User visits your PWA but then he didn’t come back. You can still reach to user by way of notifications. This will engage your end user. Notifications could be for multiple things. For example.

New content in PWA
Real time alerts
Daily trips and tricks
Updates like order update, invoice update etc.

PWA supports Web Push Notification just like native mobile apps. In this post we will explain how to add web push notification using Firebase.

Create Firebase Project

To create firebase project go to Firebase Console and login with your Google account. After you create project go to your project settings and get the sender id and add to your manifest file. Check the screen shot.

Web Push Notification Sender Id

gcm_sender_id : “YOUR_SENDER_ID”

So now we have sender id. Next step is to register for push notification.

Register for Push Notification

After your service worker is registered, you can register for push notification.

if (‘serviceWorker’ in navigator) {
navigator.serviceWorker
.register(‘service-worker.js’)
.then(registerForPushNotification);
}

function registerForPushNotification(){
navigator.serviceWorker.ready.then(function(registration) {
if (!registration.pushManager) {
alert(‘Your browser doesn\’t support push notification.’);
return false;
}

//To subscribe `push notification` from push manager
registration.pushManager.subscribe({
userVisibleOnly: true //Always show notification when received
})
.then(function (subscription) {
console.log(subscription);
var rawKey = subscription.getKey ? subscription.getKey(‘p256dh’) : ”;
var key = rawKey ? Base64EncodeUrl(btoa(String.fromCharCode.apply(null, new Uint8Array(rawKey)))) : ”;
var rawAuthSecret = subscription.getKey ? subscription.getKey(‘auth’) : ”;
var authSecret = rawAuthSecret ? Base64EncodeUrl(btoa(String.fromCharCode.apply(null, new Uint8Array(rawAuthSecret)))) : ”;
console.log(“Key: ” + key);
console.log(“AuthSecret: ” + authSecret);
console.log(‘endpoint:’, subscription.endpoint);
//Save Data on Server.
})
.catch(function (error) {
alert(‘Oops, you have disabled push notificaitons. I will not be able to give you real time updates.’)
});
});
}

function Base64EncodeUrl(str){
return str.replace(/\+/g, ‘-‘).replace(/\//g, ‘_’).replace(/\=+$/, ”);
}

function Base64DecodeUrl(str){
str = (str + ‘===’).slice(0, str.length + (str.length % 4));
return str.replace(/-/g, ‘+’).replace(/_/g, ‘/’);
}

Above functions will subscribe user for push notifications. Key, AuthSecret and endPoint, you have to save on server so that later you can use it for sending push notifications from server to your PWA application.

Also before subscribe, you can also check if user is already subscribed for notification.

registration.pushManager.getSubscription()

If user is already subscribed then you will get subscription object and just update it on server.

Here in this blog I have mentioned how to send push notification using NodeJS script.

Send Firebase Push Notification with NodeJs

Once you get notification, you have to add listener to check for notification and show it to user.

For that you have to add event listener in your service worker.

self.addEventListener(‘push’, function(event) {console.log(event);
var data = event.data.json();

var title = data.title;

var body ={
‘body’: data.message,
‘tag’: ‘PUSH’,
‘icon’: ‘images/icons/icon-32×32.png’
};

event.waitUntil(
self.registration.showNotification(title, body)
);
});

Above code will show notification in default window.

Progressive Web App – Service Worker

Leave a reply

Welcome to Progressive Web App Series. If you have missed previous blog posts, please read from following links. In this blog post we will understand one of the important component of PWA, which is Service Worker, which is nothing JavaScript file that take cares of caching, showing push notifications in PWA

What is Service Worker?

Service Worker

I am sure most of the developers are aware of concept threading. A thread is a process that can run in background without affecting the main process. Service worker is just like a thread for your PWA. It runs separately and independently from main browser thread. That means it runs in background and will be alive in browser in case you have closed your web application.

In your PWA application, a service worker is a simple JavaScript file where we can first register it, install it and activate it. Once it’s activated it will stay there till we remove it.

A Network Proxy

It can work as network proxy as well. It can intercept network requests and if needed it will block it and give you data from cache or it can let the request proceed and get you fresh data.This also gives offline capability to application. In case there is no network, we can intercept request and load data from cache. It is mainly used for showing web push notifications in PWA as it runs in background so it can subscribe for push notification events and we can easily handle push notification with it. Caching resources will make content load faster under most network conditions.

Restrictions of Service Worker

Since it runs in background, it does not have access to localStorage. It also can not access DOM, to communicate with DOM it post the messages. It can only work on HTTPS.

How to register it in PWA?

You have to create a service-worker.js file and first of all register it in your app file.

navigator.serviceWorker
.register(‘./service-worker.js’)
.then(function(){

console.log(‘Service worker registered’);

})
.catch(function(error) {

console.log(‘Service worker registration failed, error:’, error);

});

Scope of it will be defined based on where you put the file in your web app. If you put it in root folder, it can control all the files and can intercept network requests from all the file.

Installation of Service Worker

Once it’s registered and it browser treat it as new service worker, it will try to install it, you can also listen for the install event by adding install event listener in your service worker.

self.addEventListener(‘install’, function(e) {
console.log(‘[ServiceWorker] Install’);
});

Activation

Once service worker is installed, it will be activated and now it is ready to intercept network requests and caching. We can also add event handler for it.

self.addEventListener(‘activate’, function(e) {

console.log(‘[ServiceWorker] Activate’);

});

You can learn more about it on Google Developer Website.

Introduction to ServiceWorker

Create Python Webhook for Chat BOT

Leave a reply

As we all know chatbot is getting popular day by day. There are number of tools and platforms that support building chatbot.In this post we will see how we can create simple python webhook to work with bot platforms. For example,

Facebook Messenger
Slack
Telegram

These all platform provides front end for your chatbot. The frontend where user can type and see the messages from chatbot. But behind the scene there is something called webhook that powers your Chatbot. Read my blog about why Chatbots are so popular.

Why Chatbot is required?

What is a Python Webhook ?

A WebHook is an HTTP callback . An HTTP POST that occurs when something happens. For chatbot webhook is constantly running process which will receive data from bot, process it and send result back.

When you are working with platforms like FB messenger or slack for chatbot, you have to create webhook and add it to platform. This webhook will listen for users message. As soon as it get message, it will process and send reply back to bot.

Python Webhook

For this post, I assume that you have Python installed in your system.

Install virtual environment

sudo pip install virtualenv

Create folder and activate virtual environment

mkdir DemoPythonWebhook

cd DemoPythonWebhook

source bvenv/bin/activate

Install Python Flask and other packages

pip install flask

sudo pip install requests

pip install json

sudo pip install CORS

Here requests are required for sending HTTP request from webhook, if required. JSON is not encode and decode json data. CORS are required in case you want to handle cross domain requests.

Create Webhook file

Now create new file with name webhook.py and add following code to it.

import requests
import json
from flask import Flask, request
from flask_cors import CORS

app = Flask(__name__)
CORS(app)

@app.route(‘/’, methods=[‘POST’])
def handle_incoming_messages():
data = request.json
message = data[‘message’]
if(message == ‘Hi’):
return json.dumps({
“reply” : “Hi, How are you?”
})
if __name__ == ‘__main__’:
app.run(host=’0.0.0.0′)

As you can see in above code, we have created flask app and configured it for handling cross platform request. We have specified app route.

@app.route(‘/’, methods=[‘POST’])

and here we have defined a method to handle in. In this method we are decoding JSON payload and sending appropriate response back.

To run this web hook, you can type following code in terminal

python webhook.py

That’s it and your webhook is running. You can use Ngrok to publish it or you can host it on AWS lambda. You can check following link on how to use Ngrok

How to use Ngrok?

Please note this is just an example code. In case you are working with FB messenger or any other platform, you have to change code according to platform, you are using. Here are some resources in case you want to work on chatbot.

Facebook Messenger Platform for Bot

Slack Bot API

Progressive Web App – App Manifest

1 Reply

Welcome to Progressive Web App Series. If you have missed previous blog posts, please read from following links. In this blog post we will understand one of the important component of PWA, which is App Manifest, which is nothing but meta data about app.

What is App Manifest?

The manifest allows developer to configure look and feel of PWA. For example if user saves it on home page, the icon will be displayed, that we can configure it in manifest. If you don’t add this file to PWA, user will not be able to save it on home page.

So you can configure app name, app icon, launch url etc. in this file. So user can distinguish it from other sites and apps. It’s JSON based file and following is the default structure of the file.

App Manifest

{
“name”: “PWA Demo”,
“short_name”: “PWA Demo”,
“icons”: [{
“src”: “images/pwaDemo.png”,
“sizes”: “192×192”,
“type”: “image/png”
}],
“start_url”: “/index.html”,
“display”: “standalone”,
“background_color”: “#3E4EB8”,
“theme_color”: “#2F3BA2″
}

After you create this file in your PWA, you have to link this to your html page. So browser will know about it and will use it’s metadata when users saves it on home page. Add following line to your head tag.

As you can see the structure of the manifest file, you can add title, short title and add number of icons. You can add different size of the icons for different mobile screens. For example for low resolution, you can add small icon.

{
“src”: “images/pwaDemo128.png”,
“type”: “image/png”,
“sizes”: “128×128”
}

Auto Generated Splash Screen

If you add above file in PWA and save it on home screen and open it, you will notice a splash screen coming up. That’s the magic of PWA. This splash screen is built with following things of manifest.

background_color
icon
name
theme_color

This splash screen will be displayed till its loading cached resources and other resources required to start the app.

The display property define the look and feel of app. If you set it as standalone, your app will be displayed in full screen and it you want top url bar to be displayed, you can set it as browser.

So by using this file, you can configure look and feel of your PWA app. In next post we will understand about service worker. You can also get more information about manifest from Google developers site.

Web Application Manifest

Microservice Architecture with Laravel and Lumen

3 Replies

Recently in Google DevFest Ahmedabad I got to learn about Microservice Architecture. So I decided to try and implement with Laravel. In this post I am going to explain about it.

What is Microservice Architecture?

It is a method of developing software applications as a suite of independently deployable, small, modular services in which each service runs a unique process and communicates through a well-defined, lightweight mechanism to serve a business goal.

So as definition suggest, it’s a set of independently deployable services. So how we can implement it with Laravel? Please check the diagram below.

Microservice Architecture

Here first layer is a Laravel application, which act as API gateway. All the front ends call API Gateway URL, all the users are first authorized by API gateway. API gateway then calls respective service which is created in Laravel Lumen.

Gateway has sort of service repository which is nothing but singleton class which includes service metadata and hostnames. Using which it can determine which service to call.Here we are using Laravel Passport to authenticate and call service.

Gateway makes asynchronous requests to microservice which is nothing but API call. We don’t block any user request. Gateway can also modify the service response if required and also can cache the request to make response faster.

This microservice will then execute and process the call and will return data in JSON format. Each service is deployed as stand alone instance and have separate host name. The reason for using Lumen for service is it has all the features of Laravel and it’s fast and easy to scale and maintain and it’s ideal for making APIs.

Benefits of mircoservice architecture

There are number of benefits of mircoservice architecture.

Because each service is focused on only a small part they can be more lightweight and much faster performance-wise.

Also since it’s just API calls. It is possible to have different languages for different service. For example there could be notification service made in NodeJs. So it’s not necessary to be only Lumen or Laravel.

Each service can be scaled depending on usage. In short could be a good architecture for large application. In upcoming post I will show you how does this really work with example code.

Send Firebase Web Push Notification with NodeJs

Leave a reply

Sending Web Push Notification using NodeJs

Recently I was exploring NodeJs for learning and I had bit tough time in figuring out how to send Firebase PushNotifications with NodeJs. In this blog, I am going explain how to do this.

What is Web Push Notification?

Web push notifications are clickable rich content messages sent to your device by a website or a web app. Push notifications are delivered to your mobile device or desktop, even when your browser is closed. These notifications can only be sent to users who have subscribed for these notifications from a specific website. As these notifications are pushed to user’s device, users don’t have to be present on the specific website to receive these notifications.

Push Notifications are best used for delivering content in a timely manner and engage users. Other common use cases include retargeting, activation and retention.

For this blog we assume that you have added Firebase registration process on server side and getting following three fields from your client app.

1) Auth Key

2) p256dh

3) Endpoint

You will get this three things when you register for web push notification with Firebase. You have to save this on server for sending push notifications.

First of all for sending web push notifications you will need to install npm module web-push

If you have not installed it yet. Please install it first.

npm install web-push –save

Now that you have it installed, use following code to send Web Push Notifications.

const webpush = require(‘web-push’);

const sub = {

“keys” : {

“auth” : “AUTH_KEY”,

“p256dh” : “P256DH”

“endpoint” : “END_POINT”

};

const params = {

title: ‘Test title’,

message: ‘Hi How are you doing?’

};

webpush.setGCMAPIKey(‘YOUR_API_KEY);

webpush.sendNotification(sub, new Buffer(JSON.stringify(params), ‘utf8’))

.then(

function (data) {

console.log(“sucesss”);

function (err) {

console.log(err);

}

)

.catch(function (ex) {

//return callback(new Error(ex), null);

}

);

That’s it and now you will be able to send PUSH notifications using NodeJs

Hope this helps you.