Apache Spark as a service -The essentials and overview of services by IBM and Databricks

Spark  provides the flexibility you need to succeed, focusing on simplifying your time to deployment, making your business users self-sufficient, and accelerating your time to value. I have discussed what spark as a service has to offer:

Introduction about Spark 
Spark is an open source analytic engine  which is used for rapid large-scale data processing in real time. Spark provides iterative and interactive processing .Spark can be regarded as alternative for mapreduce . Apache Spark can process data from data repositories, including the Hadoop Distributed File System (HDFS) and Amazon Simple Storage Service (S3). Spark can process even unstructured data too.The main advantage is that Spark can support in-memory processing and disk based processing .Spark can be  incorporated with high level programming such as Java and other languages such as Scala,R which makes it popular among data scientists to make analytical applications.Spark make use of in-memory to its full advantage where recently read data are placed in-memory allowing faster query execution.

Spark vs Mapreduce 

• SQL queries are executed much faster in Spark than Mapreduce.
• Spark runs on hundreds of nodes whereas Mapreduce can run on thousands onf  nodes.
• Mapreduce is ideal for batch processing ,Spark is ideal for real time processing since it uses in-memory storage and processing.

Spark SQL 

Spark sql is a component in Spark big data framework which allows sql queries on data.It can query structured ,columnar,tabular data in Spark.It can be integrated with other hadoop database like Hive allowing interaction with hadoop HDFS.

Spark's MLlib 

MLlib contains functionalities such as statistics, regression,classification ,filtering which are needed for machine learning and real time analysis.

Spark's Flagship

Spark flagship converts streaming of big data into data  streams where analysing, process takes place with these data streams using Spark stream module.

Spark GraphX

This component provides processing on big data graph such as changing edges , vertices in a graph dataset.

Now let us see what factors contribute Spark as a service in other words why we can consider Spark as a service.

Spark-as-a-service 
Spark is provided as a cloud based service by IBM,Databricks because of its advantages over other processing frameworks such as Mapreduce (which I discussed earlier). Companies like databricks allows you faster deployment of Sparks.Databricks eases the process such as cluster buliding and configuration. Process monitoring , resource monitoring are taken care by Databricks. IBM's Spark-as-a-service offers new API and tac and cognitikles unstructured data and follows IBM analytics on Spark on IBM bluemix.IBM DataCap Insight Cloud services delivers data science based on external data about events, people.The company said that services doesn't require deep knowledge of big data analysis.Whenever we come across IBM and cognite computing ,we think of WATSON but this IBM Datacap Insight provides a way to handle unstructured data (i.e. machine- unfriendly data).

Best ways to tackle Big data in R

Big data contains millions of data that can be processed using R.Even though R provides few packages to support big data, extra effort is needed.Map reduce algorithms can be created using R for analysing data (Refer:The Art of R Programming - O'Reilly Media).
The original data set size could increase and lead to a bigger object during the analysing process.

I have listed out best ways to handle big data in R.

• Divide and conquer 

Large datasets could be  divided into smaller subsets and then each of these smaller subsets are worked upon.And faster solutions and different strategies can be obtained through parallel processing of these smaller subsets of data.

• Memory and hardware

Every data object is stored in memory by R.Therefore for better performance, machines should have higher capacity of  memory. 8TB along with 64-bits machines are best suitable to work with R.Another way is to make use of the packages such as "ff" and "ffbase".These packages doesn't store data in memory." Scale R" provides a variety of a algorithms for analysing data.

• Incorporating programming languages like Java or c++

Sometimes, components of a program in R can be integrated with high level languages such as Java (or)  c++ for efficient and better performance. rJava combines R and Java and regarded as connection packages (refer: Advanced R development by Hadley Wickham).Renjin is an open source project with consists of altered R interpreter within JVM.Oracle R also uses R-interpreter with variety of Mathematicals functions and libraries.

Difference between Arrays and Arraylists and Best ways to print int array,byte array,array of strings,two dimensional array and also array of array in Java

Arrays for data structure which acts as a containers that holds collections of variables of same type, in other words ,size collection of elements of the same data type. Variables stored in arrays are in continues memory locations.Before the advent of arraylists , arrays were used by programmers to store huge amount of variables 
Let us first see the differences between array lists and arrays :

• Arraylists are dynamic in size where the size of an arraylist can grow when new elements are added while arrays are fixed in size.
•  Arrays contains both objects as well as primitives whereas array list contains only objects.
• Loops such as For loops are used iterate elements and iterators can be used in arralylists.
• Arraylists ensures type safety through generics whereas arrays are homogeneous.
• Elements are added by assignment operators and add() method is used in Arraylists to add elements. 
• Arrays are mutli-dimensional where as Arraylists are single dimensional.

In order to consturct an array , you should new , then type of array and then specify the size of that array using open and closed  brackets. For example :
new int[6] //here type of array is int and size is 6.

Now let us see the different ways to print int array,byte array,array of strings,two dimensional array and also array of array.

• Printing int array : 

We can use Arrays.toString(int array), for example : 

int [] odd = {3,5,7};

System.out.printing("odd numbers are"+Arrays.toString(odd));

• Printing byte array :

Converting String into a byte array is commonly used employed in Java Cryptography Extension encryption. To convert byte array into string , create a string object and delegate the byte array to it.For example : 

String example="hello";

bytes[] bytes = example.getBytes();

String s = new String(bytes);

System.out.println("text decrypted:"+s);

• Print an array of strings :

There are two ways of printing array of strings. One way is use a FOR loop, for example:

String [] cars=new Strings[3];

cars[0]="Ferrari";

cars[1]="maruthi";

cars[2]="Datsun";

for(int i=0;i<cars.length;i++)

{

System.out.println(cars[i]);

Another way is to use Arrays.toString().

• Printing two dimensional array:

Here we use Arrays.deepToString, for example: 

String [][] greetings={{"hi","good morning "},{"hello"," good evening "}};

System.out.println(" planets"+Arrays.deepToString);

• Printing array of array:

Array of same type can be stored into another array. Arrays.deepToString is used to print array of array , for example :

String []arr1=new String [] {"hi","hello"};

String []arr2=new String [] {"how are you"};

String [][] arrayOfArray=new String[][]{arr1,arr2};

System.out.println(Arrays.deepToString));

Difference between arguments and parameters in Java

Parameter is defined in the method header whereas an argument is the instance passed to the method during run-time.

public class Parameters and Arguments {

    public static int divide(int a, int b) { //a, b are parameters here
         a=5;

         b=5;

        return a%b;

    }

    public static void main(String[] args) {
        int x=divide(a, b); //a, b are arguments here
        System.out.println(x);
    }

}

Here a and b are formal parameters which is declared in the method's header. 

Whereas the a and b becomes arguments in the point of invocation.

Possible reasons for outOfMemory error and run time and compile time errors in java

outOfJava Heap space : Java is allowed for limited usage memory. Java is divided into two regions namely, Permanent generation and Heap Space. The size for these two regions are set by JVM. OutOfMemory occurs when you add more data to heap space when it is full.
Reasons :

• Memory leaks: When you don't specify the memory by yourself. The JVM uses the Garbage collection (GC), here the unused items are cleared in memory and again made ready in other words it automatically checks for unused items and removes them.Memory leaks occurs when GC fails to recognize the unused items and fails to remove them.Thus the java heap space increases indefinitely.

Compile time error -Occurs when the code does not follow the Java semantics and syntactic rules

•  a class tires to extend more than one class 
• overloading or overriding is not correct
• referring to a out scope variable
• inner class has the same name with enclosing class name
• when class is not abstract but the methods in it are abstract
• a private member of class A is referenced by  another class B
• when creating an instance of an abstract class
• when change the value of the final member
• when two class or instance have same name
• missing brackets
• missing semicolons
• access to private fields in other classes
• missing classes on the classpath (at compile time)

Runtime error -

• using variable that are actually null (may cause NullPointerException)
• using illegal indexes on arrays
• accessing ressources that are currently unavailable (missing files, ...)
• missing classes on the classpath (at runtime)

Hash table in java

Hashtables are efficient implementation of array data structure (an associative array) that stores key/value pairs and searched by the key value.It uses a floating-point value, a string, another array, or a structure as the index. A hashtable has 2 elements, a key set and a value set.And find a way to represent and keys should always map to the appropriate.Next use a hash function that is perfect for you.A hash function depends upon the criteria, could be anything.

C language is not provided with keyed arrays , in order to access an element is by its index number.

Eg:- students[66];

I have provided the hashtable code in java :

public class HashEntry {

      private int key;

      private int value;

      HashEntry(int key, int value) {

            this.key = key;

            this.value = value;

      }     

      public int getKey() {

            return key;

      }

      public int getValue() {

            return value;

      }

}

public class HashMap {

      private final static int TABLE_SIZE = 100;

      HashEntry[] table;

      HashMap() {

            table = new HashEntry[TABLE_SIZE];

            for (int i = 0; i < TABLE_SIZE; i++)

                  table[i] = null;

      }

      public int get(int key) {

            int hash = (key % TABLE_SIZE);

            while (table[hash] != null && table[hash].getKey() != key)

                  hash = (hash + 1) % TABLE_SIZE;

            if (table[hash] == null)

                  return -1;

            else

                  return table[hash].getValue();

      }

      public void put(int key, int value) {

            int hash = (key % TABLE_SIZE);

            while (table[hash] != null && table[hash].getKey() != key)

                  hash = (hash + 1) % TABLE_SIZE;

            table[hash] = new HashEntry(key, value);

      }

}

Best Java Tools For Every Java Programmers

Java is  an Object-oriented language,robust and mainly employed for creating web applications, server handling, user-end API development. The use-case and importance of Java is huge. In-order to master java practice is vital.There are many tools but I have listed most important tools.

• Spring Framework:-The Spring Framework provides a comprehensive programming and configuration model for modern Java-based enterprise applications.

• Hibernate:- Its an object-relation mapping framework.

• JClarity:-Mainly used for monitoring performance.

• SparkJava:- Light weight web application framework.

• Thymleaf:-Java XML/XHTML/HTML5 template engine.

• JSF and JSP.

• Grails:-Web application framework.

• Elastic search:- provides tools to integrate existing applications to power their interactions with incoming data

                                                                                                                               R.Udendran M.tech cse