The WebSphere Notes

The WebSphere Notes (  is a blog that has my study notes about WebSphere Application server administration and WebSphere Portal Server developer and administration certification.


Sunil has been in the IT industry for 10 years, worked with IBM Software Labs and was part of WebSphere Portal Server Development team for 4 years, and is now working for Ascendant Technology. Sunil has been working with WebSphere Portal since 2003. He is author of "Java Portlets 101" book and more than 25 articles and has a popular blog about portlet development and administration (

Letzte Blogeinträge

  • Read configuration from .properties file in

    Montag, 23. November 2015

    This is small utility function in Scala, that takes fully qualified path of the properties file, and converts it into Map and returns. I use it for taking path of the properties file in my standalone Scala program and load it into Map

    def getConfig(filePath: String)= {    Source.fromFile(filePath).getLines().filter(line => line.contains("=")).map{ line =>      println(line)      val tokens = line.split("=")      ( tokens(0) -> tokens(1))    }.toMap  }
  • Flume Hello World tutotiral

    Montag, 28. September 2015

    I am using flume for some time now and really like it. This is simple HelloWorld tutorial that i thought would be helpful if you want to get started with Flume. This tutorial will walk you through steps for setting up Flume that listens to messages on port 44444, once it gets message it just prints it out on console, Follow these steps

    1. First create file on your machine like this

    # example.conf: A single-node Flume configuration# Name the components on this agentagent1.sources = netcat1agent1.sinks = logger1
  • Configuring Flume to write avro events into HDFS

    Freitag, 21. August 2015

    Recently i wanted to figure out how to configure Flume so that it can listen for Avro Events and whenever it gets event it should dump it in the HDFS. In order to do that i built this simple Flume configuration

    # example.conf: A single-node Flume configuration# Name the components on this agentagent1.sources = avroagent1.sinks = logger1agent1.channels = memory1# Describe/configure the sourceagent1.sources.avro.type = avroagent1.sources.avro.bind = localhostagent1.sources.avro.port = 41414
  • Configure Flume to use IBM MQ as JMS Source

    Donnerstag, 20. August 2015

    Recently i had a requirement in which i wanted to figure out how to read XML documents stored as message in IBM MQ and post them into Hadoop. I decided to use Apache Flume + Flume JMS Source + Flume HDFS Sink for this. I had to use following steps for this setup. Please note that i am not WebSphere MQ expert so there might be a better/easier way to achieve this.

  • Moving data from Avro to ORC files

    Samstag, 16. Mai 2015

    In the Importing data from Sqoop into Hive External Table with Avro encoding i blogged about how to sqoop data from RDBMS into Hive. But i wanted to take it to next step by moving the data downloaded to ORC table. I followed these steps to achieve that

    1. First thing is to find out the schema of the table in Avro and you can get that by executing following statement in hive

    show create table CUSTOMER;
  • Importing data from Sqoop into Hive External Table with Avro encoding

    Freitag, 15. Mai 2015

    I wanted to figure out how to import content of RDBMS table into Hive with Avro encoding, during this process i wanted to use external hive tables so that i have complete control over the location of files. First i did create following table in the mysql database which is on the same machine as that of my HortonWorks Sandbox

    1. First create CUSTOMER table like this in mysql

    CREATE TABLE CUSTOMER (       contactid INTEGER NOT NULL ,       firstname VARCHAR(50),       lastname  VARCHAR(50),       email varchar(50) );
  • Running oozie job on Hortonworks Sandbox

    Samstag, 2. Mai 2015

    In the Enabling Oozie console on Cloudera VM 4.4.0 and executing examples i blogged about how to run oozie job in Cloudera Sandbox. It seems this process is little bit easier in HortonWorks 2.2 sandbox. So first i had brand new HDP 2.2 image and i tried running oozie example on it by executing

    oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/ -run...
  • Exporting data from Hive table to RDBMS

    Mittwoch, 22. April 2015

    In the Importing data from RDBMS into Hadoop using sqoop i blogged about how to import data from RDBMS to Hive, but now i wanted to figure out how to export data from Hive back to RDBMS, Sqoop has export feature that allows you to export data from Hadoop directory(CSV files in a directory) to RDBMS, I wanted to try exporting data from sqoop so first i created a simple contact_hive table and populated some data in it, then i used sqoop to export the content of contact_hive table into...

  • ERROR - ERROR 1070: Could not resolve

    Freitag, 9. Januar 2015

    I was trying out Pig UDF samples from Hadoop definitive guide. Every time i tried executing com.hadoopbook.pig.IsGoodQuality UDF like this i got ERROR - ERROR 1070: Could not resolve com.hadoop.pig.IsGoodQuality using imports: error

    grunt< filtered_records = filter records by temp != 9999 and com.hadoop.pig.IsGoodQuality(quality);
  • Create ElasticSearch cluster on single machine

    Mittwoch, 31. Dezember 2014

    I wanted to figure out how to create a multi-node ElasticSearch cluster on single machine.