How to create dynamic schema in mongodb

Sometime what we need in our application is to create mongodb schema for collection and some of our collection schema are same but names are different. so everytime you are creating a new model class for the mongoose object. so this post will help you to create a mongoose object with dynamic collection name. so let me show you how to create dynamic schema in mongodb.

/* global mongoose */
var moment = require('moment');
let kindleModels = {};

/**
 * @author Wantcode.in
 * Kindle model is like store of all kind of mongo Schema
 * You don't need to create every time new schema for the mongolog
 * It's purpose is to create dynamic collection name
 */

const kindleSchema = new mongoose.Schema({
  url: String,
  method: String,
  request_type: String,
  request_data: Object,
  response_data: Object,
  userIp: String,
  fkey: {
    type: String,
    required: false,
    default: null
  },
  created_at: { type: Date, default: moment().tz(process.env.TZ).format() },
  updated_at: { type: Date, default: moment().tz(process.env.TZ).format() }
});

const kindleModel = collectionName => {
  if (!(collectionName in kindleModels)) {
    kindleModels[collectionName] = mongoose.model(
      collectionName,
      kindleSchema,
      collectionName
    );
  }
  return kindleModels[collectionName];
};

module.exports = kindleModel;

I have created a model called kindeModel this is dynamic mongoose model where you defined your collection name dynamically and call it where you want.

How to call it. import it where you want it to call.

const pageName = {
url : 'something',
fkey : 'something',
method: 'something',
user_agent:'something',
user_ip: '127.0.0.1',
request_data: {}
};      

const Kindle = kindleModel('api_logger'); // pass dynamic collection name here
const kitty = new Kindle(pageName); // pass object here
 
kitty
.saveAsync()
.then(result => {
  this.logId = result._id;
  resolve(result._id);
})
.catch(e => {
  reject(e);
});

Now your object is successfully saved in api_logger collection. Please like and share with your friends if you find this post awesome.

slack webhook nodejs tutorial

Slack has introduced ways to manage and send information via app. This method is still accurate and there hasn’t been any report regarding deprecating it anytime soon.

Slack have a very feature rich integration layer via smooth to set up WebHooks. It’s is even available to have a fully interactive message flow that deliver actions directly from the Slack chat to your application and also updates the messages depending on the return from your server.

This post will tell you how to send some custom error message or info messages to slack for notifications or history. slack webhook nodejs help you in how to create slack webhook and how to integrate it with nodejs helper.

Lets start the slack webhook api integration.

Three things you need it before this.

• Nodejs express project setup
• Slack package for nodejs
• Slack webhook

So for setting up nodejs express project i'm going into detail you can check my other article to use nodejs+express setup. I'm supposing that you already had a setup of project.

Second thing you need is slack npm package.

npm install winston winston-slack-webhook-transport --save

Third thing you need is slack webhook url . So please follow this weblink to generate webhook.

slack webhook integration

3.1 Add a new configuration so click on it to create new webhook. You also need to specify a channel or create a new one.

3.2 After that you need to specify your channel and create webhook.

After adding on incoming webhook it will give you a weblink save the link for further use.

Now lets come to the coding part. Now we are creating a helper function using winston logger . winston logger is provide you many transport hook like file logging, console logging and we are using slack logging feature.

Let's create notify.js in your nodejs project.Put your webhook in webhookUrl variable.

var winston = require('winston');
const { createLogger, format, transports } = require ('winston');
const SlackHook = require ('winston-slack-webhook-transport');
const moment = require ('moment');

var notify = createLogger ({
    level: 'warn',
    timestamp: true,
    transports: [
      new winston.transports.Console (),
      new SlackHook ({
        webhookUrl: 'https://hooks.slack.com/services/TJ5JXXXXXJ/BPPPPPQXGV/XXXXXXXXHRzTZqpdOtaDUzj8',
        formatter: info => {
          return {
            text: `*${process.env.name}*:*${process.env.NODE_ENV}* => ${info.level}: ${info.message}`,
          };
        },
        username: 'WantCodeBot',
      }),
    ],
});

Now everything is ready now you can import this notify.js file in your controller or your file to push your custom logs into slack.

const notify = require('../../../helpers/notify');
notify.warn(
`this api throw error \n> *request*: \n>${JSON.stringify(req.body)} *response*: ${JSON.stringify(response)}`
);

Now visit slack you will see a new notification in your channel. isn't it cool.

How to use robust queue system to process millions of messages using nodejs

Queue can solve your issue from carry heavy work from one server to distrubted workers etc.So why we need queue so there are some reason given below.

Redundancy is one of the most clear advantages to message queues.Queues help with redundancy by making the process that reads the message confirm that it completed the transaction and it is secure to remove it. If anything fall, worst case scenario, the message is persisted to store somewhere and won’t be lost. It can be reprocessed later.

• You don’t always know completely how much traffic your application is going to have. For example, at Stackify we collect billions of messages a weekly.
• Batching is a great conclude to use message queues. It is much more adequate to insert 500 records into a database at a time instead of 1 at a time, 500 times.
• Queues are an excellent approach to implement an asynchronous programming pattern.
• If 500 people are putting an order on your site at one time, that could create some problems with concurrency and ensuring that the first order in finishes first.
  

we are using the nodejs as our backend language and bull.js package for the queue.Bull is a Node library that implements a fast and robust queue system based on Redis.It is probably the best solution for Node.js if you want to utilize a queue mechanism.A bull queue instance can have three different roles: A job producer, a job consumer, and/or an events listener.

• producers: a Node program that adds jobs to a queue
• consumer: a Node program that defines a process function, the job content
• events listener: you can just listen to events that happen in the queue and handle different events differently.

Let's create a producers function to add the data into the queue. when the data is succesfully added to the queue. A consumer will process that data automatically. Before this you have to install the redis on your machine.

using docker :

docker run -d -p 6379:6379 -v /home/your-machine/docker/redis-volume:/data --name wantcode-redis redis

please change /home/your-machine/docker/redis-volume to your machine path where you need to store persistent data for redis.

docker start wantcode-redis
docker exec -it wantcode-redis sh (logged into shell)

Let's create a helper file where we add queuePublish function. queuePublish function is used to add into the queue. There is default setting on queue like delay how much time delay you want to added into the queue and attempts in case of failure how many times the queue retry to push. I'm using the twig template for sending email/sms using default twig sample.so ignore if you don't need it to use. queue.add is the main function to add into queue. so how queue object will get so let me show you how to create object instance for pushing into queue.

bullHelper.js

/* global TWIG_TEMPLATE */
const Twig = require('twig');
var fs = require('fs');
const commonHelper = require(`${HELPER_PATH}commonHelper.js`);
const apiConfig = require('../autoload');

module.exports = {
  queuePublish: async function (options, type = 'email') {
    var settings = {
      delay: 10000,
      attempts: 2,
      // removeOnComplete: true,
      // jobId: options.job_id,
      timeout: 60000,
      backoff: 5000 // static 5 sec delay between retry
    };
    var data;
    if (options.type == 'sms') {
      data = {
        template: options.template,
        mobile: options.mobile,
        type,
        template_option: typeof options.template_option !== 'undefined'
          ? options.template_option
          : {}
      };
    } else {
      data = {
        template: options.template,
        // id: options.job_id,
        subject: options.subject,
        email: options.email,
        type,
        template_option: typeof options.template_option !== 'undefined'
          ? options.template_option
          : {}
      };
    }

    await queue.add(
      data,
      settings
    );
  },
  prepareTemplate: function (templateName, options = {}) {
    var content = fs.readFileSync(`${TWIG_TEMPLATE}${templateName}`, 'utf-8');
    let contentRender = Twig.twig({
      data: content
    });
    let html = contentRender.render(options);
    return html;
  },
  sendEmail: async function (subject, message, emails) {
    const sendData = {
      subject,
      message,
      'emails[]': emails
    };
    const data = await commonHelper.sendXhrMultiformRequest(
      sendData,
      apiConfig.fintechMail,
      true
    );
    return data;
  },
  sendSms: async function (mobile, text = '') {
    
  }
};

Let's create queue instance object for global access so make sure you have to make a global connection for queue object.

var queue
function queueConnection() {
  queue = new Queue('mailer', {
    redis: { port:'6379', host: '127.0.0.1' }
  })
  queue
    .on('error', function (error) {
      console.error(`Error in bull queue happend: ${error}`);
    })
    .on('failed', function (job, error) {
      console.error(`bull was failed with reason: ${error}`);
    });
  global.queue = queue
}

Now the queue object will use everywhere in your application because we defined it globally.

Now make your sample file where you can call bullHelper.js queuePublish method to add into queue.

  let sms = {
    mobile: '890xxxx635',
    text: 'welcome to my website hello user'
  }
  let data = queuePublish(sms, 'sms');

Now you are successfully pushed into the queue lets make a queueHelper for reading the data from the redis .

queueHelper.js

const EventEmitter = require('events');
EventEmitter.defaultMaxListeners = 50;
const { prepareTemplate, sendEmail, sendSms } = require('../helpers/bullHelper');
// console.log(prepareTemplate('insta-case-approval.twig'));
const handleFailure = (job, err) => {
  if (typeof job !== 'undefined' && job.attemptsMade >= job.opts.attempts) {
    logger.error(
      `Job failures above threshold in ${job.queue.name} for: ${JSON.stringify(job.data)}`,
      err
    );
    job.remove();
    return null;
  }
  if (typeof job !== 'undefined' && typeof job.queue !== 'undefined') {
    logger.error(
      `Job in ${job.queue.name} failed for: ${JSON.stringify(job.data)} with ${err.message}. ${job.opts.attempts - job.attemptsMade} attempts left`
    );
  }
};

const handleCompleted = job => {
  if (typeof job !== 'undefined' && typeof job.queue !== 'undefined') {
    logger.info(
      `Job in ${job.queue.name} completed for: ${JSON.stringify(job.data)}`
    );
    job.remove();
  }
};

const handleStalled = job => {
  if (typeof job !== 'undefined' && typeof job.queue !== 'undefined') {
    logger.error(
      `Job in ${job.queue.name} stalled for: ${JSON.stringify(job.data)}`
    );
  }
};

const queueHelper = queue => {
  queue.process(5, async (job, done) => {
    try {
      var data;
      const tempOption = typeof job.data.template_option !== 'undefined' ? job.data.template_option : {};
      var getTemplate = await prepareTemplate(job.data.template, tempOption);
      if (job.data.type == 'sms') {
        data = await sendSms(job.data.mobile, getTemplate);
      } else {
        data = await sendEmail(job.data.subject, getTemplate, job.data.email);
      }
      done(null, data);
    } catch (error) {
      logger.error(
        `Job in ${job.queue.name} completed for: ${JSON.stringify(job.data)} and email response is ${data}`
      );
      done(error);
    }
  });

  // globally on complete
  queue.on('global:completed', job => {
    handleCompleted(job);
  });
  queue.on('global:stalled', () => {
    handleStalled(queue);
  });
  queue.on('global:failed', job => {
    handleFailure(job);
  });

  queue.on('global:error', job => {
    handleFailure(job);
  });
  queue.on('global:resume', function (data) {
    console.log('data : ', data);
    // queue is paused now
  });
  queue.on('global:removed', function (data) {
    console.log('data removed : ', data);
    // queue is paused now
  });
  queue.on('global:cleaned', function (data) {
    console.log('data cleaned : ', data);
    // queue is paused now
  });
};
module.exports = {
  queueHelper
};

Call the queue helper where you listen to the port. ex server.js

const { queueHelper } = require('./helpers/queueHelper');

http.listen(config.port, config.host, function () {
  console.log('HTTP MODE: ' + process.env.MODE_HTTPS);

  var message = config.message.portMsg + config.port
  console.log(message)
  queueHelper(queue);
})

Now when your application is starting on to the port the queue server is also listening you can start your queue to different port also. but for now i'm using the same application port of nodejs.

Now there is one question coming in your mind how to see the queue is added into the redis or into bull how to check it. so there is web UI which support by bull.js to see the actual status of queues.

lets install arena using docker :

docker run -p 4567:4567 -v /home/your-machine/docker/arena/index.json:/opt/arena/src/server/config/index.json mixmaxhq/arena

when you pushed your queue to publish function the queueHelper process function will automatically read it from the delayed queue. so the status of your queue you can check it from the arena dashboard. I hope you like the article so please follow our blog and like us on facebook.

Deploy javascript application to sever using shipit deployment

when you are new to some language so after completing the project.The question comes to your mind how should i deploy this project on server. The deployment should be easy enough if there is issue coming in production end so you can easily rollback the whole code.

javascript application deployment using shipit

So this post will help you deploy your javascript code using rollback feature.

Install Shipit js

npm install --save-dev shipit-cli
npm install --save-dev shipit-deploy 

Now you have to create a shipitfile.js

// shipitfile.js
module.exports = shipit => {
  // Load shipit-deploy tasks
  require('shipit-deploy')(shipit)
  require('shipit-shared')(shipit)
  require('shipit-submodule')(shipit);

  var currentPath = "/var/www/tmp/shipit-frontend-deploy";

  var Slack = require('slack-node');
  var webhookUri = "https://hooks.slack.com/services/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";
  var slack = new Slack();
  slack.setWebhook(webhookUri);

  shipit.initConfig({
    default: {
      workspace: currentPath,
      //workspace: '/tmp/shipit-folder-name',
      //updateSubmodules: true,
      deployTo: '/var/www/frontend-deploy',
      repositoryUrl: 'git@bitbucket.org:imsonujangra/frontend.git',
      // shared: {
      //   dirs: ['node_modules'],
      //   overwrite: true,
      //   triggerEvent: "updated"
      // },
      keepReleases: 2,
      rsync: ['--del'],
      shallowClone: true,
      submodules: true,
      ignores: ['.git', 'node_modules']
    },
    development: {
      branch:'master',
      servers: 'groot@192.169.12.222',
    }
  })

  shipit.blTask('submodules', async () => {
    shipit.log('Starting...');
    await shipit.local(`cd ${shipit.workspace} && git clone git@bitbucket.org:imsonujangra/react-common.git`)
  });

  shipit.blTask('npm:install', async () => {
    await shipit.remote(`cd ${shipit.releasePath} && npm install && npm link gulp && npm run gulp-default:local && npm run build:local`)
  })

  shipit.blTask('server:start', async () => {
    const command = 'chmod +x frontend.sh && ./frontend.sh'
    await shipit.remote(`cd ${shipit.currentPath} && ${command}`)
  })

  shipit.blTask('server:restart', async () => {
    const command = 'forever restartall'
    await shipit.remote(`cd ${shipit.config.deployTo} && ${command}`)
  })

  shipit.blTask('server:copyConfig', async () => {
      shipit.log('copying server file :: >>>>> ');
      await shipit.remote(`cp -r /var/www/common_shared/frontend.sh ${shipit.releasePath}/`);
  })

  shipit.on('updated', () => {
    shipit.start('server:copyConfig')
    
  })

  shipit.on('fetched', () => {
    shipit.start('submodules');
    
  })

  shipit.on('published', () => {
    shipit.start('npm:install');
  })

  shipit.on('deployed', function () {
    shipit.start('server:start')
  });

  shipit.blTask('slack', function(cb){
    var workspace = shipit.config.workspace;
    shipit.local('git rev-parse HEAD', {cwd: workspace}).then(function(res) {
      let pack = {
        "name":"frontend-deploy",
        "version":"1.0.1",
        "url":"https://bitbucket.org/imsonujangra/frontend/commits/"
      };
      var githubLink = pack.url.replace('/issues',`/commit/${res.stdout}`).trim();
      slack.webhook({
            username: "Shipit",
            text: `${pack.name} v${pack.version} - Deployed to ${shipit.environment} \n#${res.stdout}\n<${githubLink}|View on GitHub>`
            }, function (err, response) {
              return console.error('upload failed:', err);
            }
      );
    });
  });

  shipit.on('cleaned', function () {
    shipit.start('slack')
});
}

For complete article you can go to my medium link. I have already written the whole article there so click here  

 Don't forget to subscribe these blog for latest technologies article.