Installing ElasticSearch on Amazon EC2

I wanted to set up ElasticSearch on Amazon EC2 and these are the steps that i followed to do that
  • Connect to your Amazon EC2 instance using SSH and execute following command to download ElasticSearch v 90.9 on my instance
    It took less than a minute to download the elasticsearch binary on the instance
  • Next i executed following command to unzip the elasticsearch-0.90.9.tar.gz
    tar -xf elasticsearch-0.90.9.tar.gz
  • After extracting the elasticsearch binaries start it by executing following command
    elasticsearch-0.90.9/bin/elasticsearch –f
  • After starting the ElasticSearch instance i could use the ES REST API from the SSH console on the same instance
    curl -XGET http://localhost:9200/_search
  • But since i want to access it from my local machine/outside the instance i had to open the 9200 port on that instances firewall that i could do by changing the security group and adding 9200 and 9300 ports to it
  • Then you can use the public DNS for your EC2 instance and query the REST instance from your local machine like this
    curl -XGET http://<yourinstancename>

Using Elastic Search to implement Locate US/ Geo Saptial Search

Elastic Search has nice support for geo spatial search. I wanted to try that out. So i started by creating a index called testgeo and i did add addresses of few walmart locations in it. Then i used Elastic Search to figure out what store location is close to my location. I followed these steps to implement this sample
  1. First i did create a new Index named testgeo like this, while creating index i marked location field as geo_type, this makes it possible to make geo distance queries.
    curl -XPUT 'localhost:9200/testgeo' -d '{
        "mappings": {
            "walmart": {
              "properties": {
                "shopName": {
                    "type": "string"
                 "address": {
                    "properties": {
                       "city": {
                          "type": "string"
                       "state": {
                          "type": "string"
                       "streetName": {
                          "type": "string"
                       "location": {
                          "type": "geo_point"
  2. Then i did add few locations to my index for each of the location i used the latitude, longitude for address of the location and stored it in my testgeo index
    curl -XPOST 'localhost:9200/testgeo/walmart' -d '{
       "storeName": "Walmart Supercenter - Marina",
       "address": {
          "streetName": "150 Beach Rd",
          "cityName": "Marina",
          "state": "CA",
          "zipCode": 93933,
          "location": [
    curl -XPOST 'localhost:9200/testgeo/walmart' -d '{
       "storeName": "Walmart Mkt - Modesto",
       "address": {
          "streetName": "\"1421 Coffee Rd",
          "cityName": "Modesto",
          "state": "CA",
          "zipCode": 95355,
          "location": [
    curl -XPOST 'localhost:9200/testgeo/walmart' -d '{
       "storeName": "Walmart Supercenter - Patterson",
       "address": {
          "streetName": "\"1030 Sperry Ave",
          "cityName": "Patterson",
          "state": "CA",
          "zipCode": 95363,
          "location": [
  3. Once my index data is in place i could query the index with type equal to geo_distance, and i had to give a location from which i want to search all the stores in 100 KM of the location
    curl -XPOST 'localhost:9200/testgeo/repair/_search?pretty=true'  -d '
        "query": {
            "filtered": {
               "query": {
                    "match_all": {}
               "filter": {
                   "geo_distance": {
                      "distance": 100,
                      "distance_unit": "km",
                      "address.location": {
                         "lat": 37.53,
                         "lon": -121.97
Elastic Search returned list of stores matching the criteria.


I was trying to use REST API for getting data. So i started following the document. But after i followed it i kept getting Unable to get access token exception, more details here

 at com.force.sdk.oauth.connector.ForceOAuthConnector.createTokenInternal(
2013-11-28T05:58:29.294965+00:00 app[web.1]:  at
2013-11-28T05:58:29.294965+00:00 app[web.1]: 05:58:29,267 ERROR OAuthAuthenticationProvider:82 - Unable to get access token
2013-11-28T05:58:29.294965+00:00 app[web.1]: Missing refresh token on response
2013-11-28T05:58:29.294965+00:00 app[web.1]:  at
2013-11-28T05:58:29.294965+00:00 app[web.1]:  at com.force.sdk.springsecurity.OAuthAuthenticationProvider.authenticate(
2013-11-28T05:58:29.300136+00:00 heroku[router]: at=info method=GET path=/_auth?code=aPrxkpHJh6tvoT6FsBbmu3.hEs0MRNTo0tj3P9AWu2oQ3aJeF7rXY._JA1wepFrePIgKkBuF4w%3D%3D& fwd="" dyno=web.1 connect=1ms service=446ms status=401 bytes=1083
2013-11-28T05:58:29.295172+00:00 app[web.1]:  at$VirtualFilterChain.doFilter(
2013-11-28T05:58:29.294965+00:00 app[web.1]:  at com.force.sdk.oauth.connector.ForceOAuthConnector.getAccessToken(
2013-11-28T05:58:29.294965+00:00 app[web.1]:  at
2013-11-28T05:58:29.295172+00:00 app[web.1]:  at$VirtualFilterChain.doFilter(
2013-11-28T05:58:29.294965+00:00 app[web.1]:  at com.force.sdk.springsecurity.AuthenticationProcessingFilter.doFilter(
2013-11-28T05:58:29.294965+00:00 app[web.1]:  at com.force.sdk.springsecurity.AuthenticationProcessingFilter.attemptAuthentication(
2013-11-28T05:58:29.295172+00:00 app[web.1]:  at
2013-11-28T05:58:29.295172+00:00 app[web.1]:  at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(
2013-11-28T05:58:29.295172+00:00 app[web.1]:  at org.springframework.web.filter.DelegatingFilterProxy.doFilter(
2013-11-28T05:58:29.295172+00:00 app[web.1]:  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
2013-11-28T05:58:29.295172+00:00 app[web.1]:  at
2013-11-28T05:58:29.295172+00:00 app[web.1]:  at$VirtualFilterChain.doFilter(
2013-11-28T05:58:29.295416+00:00 app[web.1]:  at org.apache.catalina.core.StandardContextValve.invoke(
2013-11-28T05:58:29.295172+00:00 app[web.1]:  at
2013-11-28T05:58:29.295416+00:00 app[web.1]:  at org.apache.catalina.core.StandardWrapperValve.invoke(
2013-11-28T05:58:29.295416+00:00 app[web.1]:  at org.apache.catalina.core.StandardHostValve.invoke(
2013-11-28T05:58:29.295416+00:00 app[web.1]:  at org.apache.catalina.valves.ErrorReportValve.invoke(
2013-11-28T05:58:29.295172+00:00 app[web.1]:  at org.apache.catalina.core.ApplicationFilterChain.doFilter(
2013-11-28T05:58:29.295416+00:00 app[web.1]:  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(
2013-11-28T05:58:29.295416+00:00 app[web.1]:  at org.apache.coyote.http11.AbstractHttp11Processor.process(
2013-11-28T05:58:29.295946+00:00 app[web.1]:  at java.util.concurrent.ThreadPoolExecutor.runWorker(
2013-11-28T05:58:29.295946+00:00 app[web.1]:  at
2013-11-28T05:58:29.295416+00:00 app[web.1]:  at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(
2013-11-28T05:58:29.295416+00:00 app[web.1]:  at$
2013-11-28T05:58:29.295946+00:00 app[web.1]:  at java.util.concurrent.ThreadPoolExecutor$
2013-11-28T05:58:29.295416+00:00 app[web.1]:  at org.apache.catalina.core.StandardEngineValve.invoke(
2013-11-28T05:58:29.295416+00:00 app[web.1]:  at org.apache.catalina.connector.CoyoteAdapter.service(
2013-11-28T05:58:24.518283+00:00 heroku[router]: at=info method=GET path=/people/ fwd="" dyno=web.1 connect=1ms service=5ms status=302 bytes=0
After some googling i found the problem was that in oAuth setting i had to add 2 scopes to Selected oAuth scopes
  1. Full Access(full)
  2. Perform request on your behalf at any time (refresh_token)
Once i set up the oAUth settings properly the sample started working

Using Salesforce SOAP API from Java Code

Recently i had to develop a portlet that talks to and retrieves some data. I decided to use the SalesForce SOAP API for that, i followed these steps
  1. First step is to get your security token, follow steps as described in Generating Security Token for post for that
  2. Second step is to login into your account and download the Enterprise .wsdl, Follow steps 1-3 from Accessing SalesForce data using SOAP Service - SoapUi post for that
  3. You should use the web services connector (WSC) which has helper code that simplify working with SOAP service.
  4. Once you have WSC project downloaded on your machine and complied into jar copy it in a directory and then also copy your enterprise.wsdl in the same directory, then execute
    java -classpath target/force-wsc-29.0.0-jar-with-dependencies.jar enterprise.wsdl enterprise.jar
    It will generate enterprise.jar file for you
  5. Now create a Java Project and add both enterprise.jar and force-wsc-29.0.0-jar-with-dependencies.jar in your classpath
  6. Last step is to create class like this Replace userId, password and securitytoken in this file with the values you have
  7. When you run the class it will display list of contacts in your account

Accessing SalesForce data using SOAP Service - SoapUi

Recently i was trying to figure out how to access SalesForce data from outside. I had two options either to use REST or use SOAP API for accessing the data. I tried both and these are my notes for accessing SalesForce Soap API using Soap client such as SoapUi.
  1. First thing that you would want to do is generate security token for salesforce. Follow these steps for generating security token
  2. Once you have security token, login into the SalesForce console to download the WSDL, Click on Develop -< API and you will see page where you can download all the WSDL's, Download the Enterprise WSDL
  3. Now create new SOAP Ui project using the Enterprise WSDL you just downloaded
    You will notice that it generates bunch of SOAP operations
  4. In order to use the SOAP API you will have to first send a login request to SalesForce and get sessionid once you have it you can use that for making subsequent calls. First call the Login service like this
    <soapenv:Envelope xmlns:soapenv="" 
    You should replace replacerwitheuserid with your userId (email id that you use for login) and also replace replacewithpassword with your password and use the security token that you got in email to replace replacewithsecuritytoken. Make a request and you will get a response which looks like this
    <soapenv:Envelope xmlns:soapenv="" 
                   <roleId xsi:nil="true"/>
                   <userDefaultCurrencyIsoCode xsi:nil="true"/>
                   <userFullName>Sunil Patil</userFullName>
    You will need the value of sessionId element and value of serverUrl in subsequent requests. so note it down
  5. Next assume that you want to execute query request that will give you list of all the contacts using SELECT Id, FirstName, LastName From Contact query, so use XML that looks like this for the soap request. Replace the replacewithyoursessionid with the sessionId that you got from the login request and also change the URL where SOAP request is made to value of serverUrl
    SELECT Id, FirstName, LastName  From Contact<
    You will get response with list of contacts

Generating Security Token for

If you want to use SalesForce SOAP API then you will need security token. If you dont have the security token SalesForce will generate Soap fault like this

<soapenv:Envelope xmlns:soapenv="" 
         <faultstring>LOGIN_MUST_USE_SECURITY_TOKEN: Invalid username, password, security token; or user locked out. Are you at a new location? When accessing Salesforce--either via a desktop client or the API--from outside of your company’s trusted networks, you must add a security token to your password to log in. To receive a new security token, log in to at and click Setup | My Personal Information | Reset Security Token.</faultstring>
            <sf:LoginFault xsi:type="sf:LoginFault">
               <sf:exceptionMessage>Invalid username, password, security token; or user locked out. Are you at a new location? When accessing Salesforce--either via a desktop client or the API--from outside of your company’s trusted networks, you must add a security token to your password to log in. To receive a new security token, log in to at and click Setup | My Personal Information | Reset Security Token.</sf:exceptionMessage>
Follow these steps to generate a security token
  1. Login into your Salesforce account
  2. Click on your use name and you will get drop down like this, click on My Settings sub menu
  3. On the next page you will get submenu on the side click on Personal side menu, once its expanded click on Reset my security token link
  4. Once your on the Reset My Security Token button and it will trigger security token generation process
  5. In few minutes it will send email to your address with security token, use that for the subsequent SOAP calls

Customizing the JSON output generated by Jackson

Recently i was using Jackson JSON Processor for converting java object into JSON. When i did that i noticed JackSon was storing date in the long format i.e. it was converting date.getTime() and using that long value in JSON instead of formatted date. This is how my Contact object looks like Note: please ignore line 32 @JsonSerialize(using = CustomDateSerializer.class) for now. I had following code to create object of Contact and converting and then using Jackson to convert it to String and write it on console This was the output that got generated

Now i wanted to customize how the date so that it got generated in dd-MMM-yyyy format. In order to do that i started by creating CustomDateSerializer class like this Then i had to change the class to add @JsonSerialize(using = CustomDateSerializer.class) annotation to getDateOfBirth() field, i could attach it to any other date field that i want. Now when i run JSONTester this the output that it generates

You can download the source code for this program from here

Using Mule to get Geo Code for address

For last few days i am playing around with Mule, its really cool integration framework. You can use it to build integration applications by using small amount of code. I had this requirement where given address i wanted to get the Geo Coding information for that address, I wanted to use Google Maps API for getting GeoCode information for the address. The way Google Maps API works is it has a REST API, it takes address as query string and it returns JSON structure. Ex. If you copy paste this URL in the browser Oak Road,Walnut Creek,CA,94597&sensor=false You will get a JSON structure with longitude and latitude embedded in lot of other information

   "results" : [
         "address_components" : [
               "long_name" : "3055",
               "short_name" : "3055",
               "types" : [ "street_number" ]
               "long_name" : "Oak Road",
               "short_name" : "Oak Rd",
               "types" : [ "route" ]
               "long_name" : "Walnut Creek",
               "short_name" : "Walnut Creek",
               "types" : [ "locality", "political" ]
               "long_name" : "Contra Costa County",
               "short_name" : "Contra Costa County",
               "types" : [ "administrative_area_level_2", "political" ]
               "long_name" : "California",
               "short_name" : "CA",
               "types" : [ "administrative_area_level_1", "political" ]
               "long_name" : "United States",
               "short_name" : "US",
               "types" : [ "country", "political" ]
               "long_name" : "94597",
               "short_name" : "94597",
               "types" : [ "postal_code" ]
         "formatted_address" : "3055 Oak Road, Walnut Creek, CA 94597, USA",
         "geometry" : {
            "bounds" : {
               "northeast" : {
                  "lat" : 37.9302338,
                  "lng" : -122.0582288
               "southwest" : {
                  "lat" : 37.9302299,
                  "lng" : -122.0582466
            "location" : {
               "lat" : 37.9302299,
               "lng" : -122.0582466
            "location_type" : "RANGE_INTERPOLATED",
            "viewport" : {
               "northeast" : {
                  "lat" : 37.93158083029149,
                  "lng" : -122.0568887197085
               "southwest" : {
                  "lat" : 37.9288828697085,
                  "lng" : -122.0595866802915
         "types" : [ "street_address" ]
   "status" : "OK"
Out of all the data returned by Google MAPS api i am only interested in the Lat and Long and i want to return it in GeoJSON format like this

Which is [lng,lat] format, some open source JavaScript Mapping frameworks understand the GeoJSON format. I built a simple Mule integration application which takes STREETLINE1, CITY, STATE, ZIPCODE as argument and returns geo location in GeoJSON format like this [lng,lat]. I followed these steps to build the application
  1. First i built a Mule Flow which looks like this
    In this flow i have a composite input source that could take input on either VM transport or HTTP transport and then uses the values submitted by user to make a request to Google Maps API. Once the response is returned it uses DataMapper to convert the big response that Google Maps API returns into simple GeoJSON response.
  2. This is how the Data Mapping file looks like
    If you look at the script of this file it looks like this
After deploying this application i can make the POST call to http://localhost:9081/geocode URL and see the location in geojson format like this

How Scala code gets compiled into Java

For last few days i am learning Scala + play framework. Since i have been using Java for long time now, one that always helps is to convert Scala code into Java Code and see what is going on under the hood. I follow these steps
  1. First i create a new Scala File say Contact.scala which has one Scala class like this
    class Contact{
       var firstName =""
       private var lastName = ""
  2. Then go to the directory that has Contact.Scala and compile it using scalac Contact.scala, The scala compiler will create .class files for every class in that .sclaa file in appropriate package. In my case i have only one class in my Contact.scala so it creates Contact.class
  3. Next use javap Contact and and i can see following output being generated on the command line
    Compiled from "Person.scala"
    public class Contact extends java.lang.Object{
        private java.lang.String firstName;
        private java.lang.String lastName;
        public java.lang.String firstName();
        public void firstName_$eq(java.lang.String);
        private java.lang.String lastName();
        private void lastName_$eq(java.lang.String);
        public Contact();

Using AngularJs in Worklight/PhoneGap application

Angular.js is a JavaScript MVC framework, that makes development of HTML applications easy. I wanted to figure out how to use it for developing Worklight application so i followed these steps to build simple Hello AngularJs application
  • First create a WorkLight application using WorkLight wizard, make sure that it works
  • Next make changes in the index.html or entry page of your application to include angular.js from Google CDN and also add <p>Hello {{'World'.length}}</p>
  • to test if AngularJs template is working
     <meta charset="UTF-8">
     <meta name="viewport" content="width=device-width, 
    initial-scale=1.0, maximum-scale=1.0, 
    minimum-scale=1.0, user-scalable=0">
     <link rel="shortcut icon" href="images/favicon.png">
     <link rel="apple-touch-icon" href="images/apple-touch-icon.png">
     <link rel="stylesheet" href="css/HelloWorld.css">
     <script>window.$ = window.jQuery = WLJQ;</script>
    <body id="content" style="display: none;" ng-app>
     <h1>Hello Angularjs</h1> 
     <p>Hello {{'World'.length}}</p>
     <script src="" />
     <script src="js/initOptions.js"></script>
     <script src="js/HelloWorld.js"></script>
     <script src="js/messages.js"></script>
After deployment you will notice that it prints Hello + length of 'world' which is 5 characters like this