satya - 9/9/2019, 5:33:49 PM

To unzip that file you need 7 zip. Download that program from here for windows

To unzip that file you need 7 zip. Download that program from here for windows

You will notice that it is a secure site (https) which means its URL is validated and you can trust what is downloaded from that site as long as the site is not compromised. Download the 64 bit version of executable.

satya - 9/9/2019, 5:36:18 PM

Common hadoop binaries from github repo: steveloughran/winutils

Common hadoop binaries from github repo: steveloughran/winutils

Windows binaries for Hadoop versions till Hadoop 2.6.0. For later releases you need to use the repo from the link below

satya - 9/9/2019, 5:37:39 PM

Link to winutils repo for later releases: cdarlint/winutils

Link to winutils repo for later releases: cdarlint/winutils

winutils.exe hadoop.dll and hdfs.dll binaries for hadoop windows

satya - 9/9/2019, 5:38:46 PM

What are these winutils?

satya - 9/9/2019, 5:39:24 PM

Setting up hadoop home and path once you have these utils

HADOOP_HOME=<your local hadoop-ver folder>
PATH=%PATH%;%HADOOP_HOME%\bin

satya - 9/9/2019, 5:47:22 PM

So download the winutils from this cdarlint path

C:\satya\i\hadoop-winutils\winutils\hadoop-2.6.5\bin
C:\satya\i\hadoop-winutils\winutils\hadoop-2.7.3\bin

C:\satya\i\hadoop312-common-bin\bin

where the hadoop_home is

C:\satya\i\hadoop312-common-bin

and the path is 

C:\satya\i\hadoop312-common-bin\bin

satya - 9/9/2019, 5:56:17 PM

Download Python from here for windows 10

satya - 9/9/2019, 6:09:30 PM

MS Document: Getting started with python in vscode is here

MS Document: Getting started with python in vscode is here

Caution: Do not follow the instructions here to install python through windows 10 apps. Instead use the python.org as above to install python. Just be careful about "pip". It may be better to try "pip" from an outside command line and not the one from inside vscode.

However this link may be useful as you get more into vscode and python for other aspects.

satya - 9/9/2019, 6:11:15 PM

Using multiple python environments in vscode

Using multiple python environments in vscode

It is a bit strange how the IDEs seem to treat python. It seems you can identify a different python interpreter for each folder that is opened in vscode. It is too early for me to comment in detail on this.

satya - 9/9/2019, 6:14:42 PM

Here is a link from python.org on working with python on windows

Here is a link from python.org on working with python on windows

There is a lot of windows specific info on python. I need to explore this more. For now leave this here as a key reference to come back to.

satya - 9/9/2019, 6:20:36 PM

PyPi.org

PyPi.org

This is where public python packages are kept such as PySpark

satya - 9/9/2019, 6:23:19 PM

Here is a link in PyPi to PySpark package

Here is a link in PyPi to PySpark package

I need more research on the relationship between python, Spark, and PySpark to speak more lucidly. For now I seem to need to install pyspark to get rid of errors in Vscode intellisense. But that happens through "pip" install which uses the pypi.org behind the scenes to pull it from.

satya - 9/10/2019, 11:06:27 AM

Java SE Development Kit 8 Downloads: from Oracle

satya - 9/10/2019, 11:14:24 AM

MIT Archive of Shakespeare sonnets text

MIT Archive of Shakespeare sonnets text

when you start using pyspark code you can use some text files to process. This URL has some text that you can copy as files and then analyze the text for words using pyspark sample code.

satya - 9/10/2019, 11:19:21 AM

5. Detailed analysis for each of the installations

5. Detailed analysis for each of the installations

satya - 9/10/2019, 11:38:01 AM

Start with JDK8

satya - 9/10/2019, 11:39:06 AM

So the paths are

JAVA_HOME to \jdk8\jre
Path: \jdk8\jre\bin

satya - 9/10/2019, 11:42:46 AM

Install Python next

satya - 9/10/2019, 11:43:58 AM

Paths for python are

\python374
\python374\scripts

//Notice there is no python_home
//as far as I know for now

satya - 9/10/2019, 11:46:30 AM

Install 7zip if you don't have it

satya - 9/10/2019, 11:52:57 AM

Download and unzip Spark

satya - 9/10/2019, 11:53:25 AM

If need be

spark_home=c:\i\spark
path to c:\i\spark\bin

satya - 9/10/2019, 12:59:21 PM

Next install Hadoop binaries

satya - 9/10/2019, 1:02:33 PM

Set the following system variable

_JAVA_OPTIONS "-Djava.net.preferIPv4Stack=true"

This is to disable ipv6. Not sure why.

satya - 9/10/2019, 1:06:55 PM

How best to set system variables in Windows 10

satya - 9/10/2019, 1:10:02 PM

Let me summarize the system environment variables one more time

#***************************************
#Java options to disable ipv6
#***************************************
_JAVA_OPTIONS "-Djava.net.preferIPv4Stack=true"
JAVA_HOME "%install-dir%\jdk8\jre"
HADOOP_HOME C:\satya\i\hadoop312-common-bin

#***************************************
#Add the following paths to the system path variable
#***************************************
#Java path
%install-dir%\jdk8\jre\bin

#python-path
%install-dir%\python374
%install-dir%\python374\scripts


#hadoop-path
%install-dir%\hadoop312-common-bin\bin

#Likely optional spark home
spark_home=%install-dir%\spark234 (what ever version u have)

#Likely optional spark path
%install-dir%\spark234\bin

satya - 9/10/2019, 1:12:37 PM

Running windows command line as an administrator

satya - 9/10/2019, 1:15:35 PM

It is time to test the installation

satya - 9/10/2019, 1:17:53 PM

There was an additional precaution in the 24 hours book to do the following

mkdir C:\tmp\hive
C:\Hadoop\bin\winutils.exe chmod 777 /tmp/hive

It may be safer to run the command line as an admin for this command. I don't remember now how I did it. I also don't see any files in there. So I don't even know if this is needed.

satya - 9/10/2019, 1:30:52 PM

7. Submitting pyspark python files using submit command to run on the local spark stand alone cluster.

satya - 9/10/2019, 1:37:25 PM

Where can I get sample pyspark files?

satya - 9/10/2019, 1:38:08 PM

Now you can use this command to submit wordcount.py to execute on spark

c:\satya\i\spark\bin\spark-submit --master local[4] wordcount.py sonnet2.txt

satya - 9/10/2019, 1:50:40 PM

8. Working with VSCode on pyspark code

satya - 9/10/2019, 1:51:57 PM

Here is an example wordcount that I had altered slightly

###############################################
# To test flatmap
###############################################

from __future__ import print_function

import sys
from operator import add

from pyspark.sql import SparkSession

#***********************************
#Function: printCollected
#***********************************


def printBeginMsg(msg):
    print ("*****************************")
    print ("*" + msg)
    print ("*****************************")

def printEndMsg(msg):
    print ("*****************************")
    print ("*" + msg)
    print ("*****************************")

def printCollected(msg, collectedRDD):
    printBeginMsg(msg)
    for item in collectedRDD:
        print (item)
    printEndMsg(msg)

#***********************************
#End of function
#***********************************

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: wordcount <file>", file=sys.stderr)
        sys.exit(-1)

    spark = SparkSession        .builder        .appName("PythonWordCount")        .getOrCreate()


    #
    # spark.read is a property
    # it returns a dataframereader
    #
    lines = spark.read.text(sys.argv[1]).rdd.map(lambda r: r[0])
    printCollected("Raw lines", lines.collect())

    lineAsListOfWords = lines.map(lambda x: x.split(' ')) 
    printCollected("Raw lines split into words. Each line is a list of words",         lineAsListOfWords.collect())

    justAListOfWords = lineAsListOfWords.flatMap(lambda x: x.lower())
    printCollected("Just A List of Words, from flatmap", justAListOfWords.collect())

    #make each word a list which is (word, length-of-thatword)
    # WordObject: (word, length, howmany)
    listOfWordObjects = justAListOfWords.map(lambda x: (x, len(x), 1))
    printCollected("List of Word Objects", listOfWordObjects.collect())

    spark.stop()

satya - 9/10/2019, 1:52:18 PM

I was getting an error at the following line

from pyspark.sql import SparkSession

satya - 9/10/2019, 1:53:12 PM

So I have decided to install pyspark through pip install

satya - 9/10/2019, 1:54:48 PM

How to install "pip install pyspark"

//the pip script is in python\bin directory
//As long as that is on the path you can run pip

//Run the following from any command line
//Safe to run the command line as an admin

//Install this first
pip install pypandoc

//Then install this
pip install pyspark

satya - 9/10/2019, 1:56:15 PM

9. How to exercise the PysSpark API

satya - 9/10/2019, 1:58:27 PM

All submits seem to start with this code

#Refer to the following package in the APIs
from pyspark.sql import SparkSession

//All starts with a SparkSession object
//Look at the API for SparkSession
spark = SparkSession        .builder        .appName("PythonWordCount")        .getOrCreate()

//This creates a new SparkSession with a name
//uses the standard build pattern

//Now you can use the SparkSession to
//read files as data frames and RDDs

satya - 9/10/2019, 2:00:14 PM

Take a look at this code

    # spark.read is a property
    # it returns a dataframereader
    # .text returns a dataframe
    # dataframe has an RDD
    # Get a new RDD that is just a collection of lines
    lines = spark.read.text(sys.argv[1]).rdd.map(lambda r: r[0])

    #Bring it back to the driver and print it
    #Because RDDs are on multiple machines up until "collect"
    printCollected("Raw lines", lines.collect())

satya - 9/10/2019, 2:00:37 PM

10. Now that you have a house, how to begin the practice of living in PySpar

10. Now that you have a house, how to begin the practice of living in PySpar

satya - 9/10/2019, 3:51:08 PM

Next steps

satya - 9/10/2019, 3:55:21 PM

More....

satya - 9/10/2019, 3:56:24 PM

Understand the following APIs ...and more

satya - 9/10/2019, 4:22:41 PM

A Shortcut to Big Data - Lesson Plan 1, Azure and Spark: Smart Programmer Series

A Shortcut to Big Data - Lesson Plan 1, Azure and Spark: Smart Programmer Series

This article goes over how to setup a Databricks spark cluster in Azure. Much like this second lesson, this first lesson at this URL goes into lot of details on how to do that