TekInvaderZ, LLC is on a lookout for Hadoop Developer who can execute technical initiatives at various client locations. The Hadoop Developer will be responsible for:

Implementing data pipelines using frameworks like Spark with Scala which are used to get the data from source, making transformations over it and loading it to a sink.
Developing Software Development Kits (SDKs) which includes advanced programming techniques using programming languages like Java and Scala which are used to parse the data from Rest API and making it available with additional data to different teams where they can leverage this SDK as a dependency on their side and make use of all the methods and functions in it.
Creating applications which are used to get the data from different sources and publish that data to an Amazon SNS (Simple Notification Service) topic over Amazon Web Services (AWS) Cloud to run visualizations over that data and get some insights which can save a lot of money to the business.
Developing, building and maintaining the data lake for the project and making the data available from multiple external sources in the formats like Parquet, Avro, RC and JSON formats according to the business requirement in the data lake.
Extensively using Amazon Web Services like EC2, S3 buckets, EMR, Lambda, SNS, Cloud Watch and Elastic search to create, configure, run and monitor the spark jobs and to store the data.
Using tools in the Hadoop framework such as Hive to store the data and query over that data using Spark SQL and Sqoop to import and export data from databases like MySQL and Oracle.
Using Apache Ranger which deals with granting policies and accesses to the users who are going to query the data from a hive server.
Working extensively on performance tuning hive queries by parallelizing the queries, using operations like cluster by and distribute by clauses. Tuning the performance of Spark jobs by choosing the right file format to make transformations over the data and techniques like Salting to overcome data skewing.

Qualification

Bachelor's degree in Computer Science, Computer ScienceInformation Systems, Information Technology, or a combination of education and experience equating to education in the United States

Hours per week:

40hrs

Employment Type:

Full Time, Long Term.

Compensation:

Depends on the location