Essential Python
satya - 8/25/2019, 11:08:40 AM
Python in VSCode is documented here
satya - 8/25/2019, 11:09:07 AM
Chose to use VSCode as I have it already.
Try not to use a separate IDE for each technology!!!
satya - 8/25/2019, 11:10:10 AM
Anaconda or windows app
Anaconda seem to be around 600MB and seem to have lots of bells and whistles such as R and data science stuff.
For starters I went with the windows python app in windows store at 60MB I believe.
satya - 8/25/2019, 11:10:38 AM
When installed and run it seem to open a python 3.7.4 interactive shell.
When installed and run it seem to open a python 3.7.4 interactive shell.
satya - 8/25/2019, 11:11:29 AM
The shell seem to suggest a tutorial
satya - 8/25/2019, 11:14:07 AM
The standard library is documented here
satya - 8/25/2019, 11:16:26 AM
High level
High level literal data structures such as lists, tuples (tables/rows/columns)
interpreted, dynamic, Scripting benefits
Object oriented
open for modules and new packages
satya - 8/25/2019, 11:16:48 AM
Python packages called pypi (python package index) is documented here
Python packages called pypi (python package index) is documented here
satya - 8/25/2019, 1:34:04 PM
len() for example is a built in function in python: The other are here
len() for example is a built in function in python: The other are here
satya - 8/25/2019, 1:34:59 PM
Built-in functions in Python
Built-in functions in Python
satya - 8/25/2019, 1:48:24 PM
Lesson 1 - Basics
# This is a comment
#This is a hello world in python
print ("hello world")
#A partial comment
msg1 = "Partial comment" #Partial comment
# Common
# Escaped strings with # single quotes and double quotes
#************************************
#Multiline strings
#End of lines are preserved unless removed with an escape \
#************************************
multilineString = """First line
Second line
third line
"""
print (multilineString)
#************************************
#Glued strings
#No need for + if next to each other
#************************************
msg2 = "test" + "another test"
msg3 = "test" "another test"
print (msg2)
print (msg3)
#You have to put them in brackets to make the following work
msg4 = ("test"
"another test string"
"and one more string")
print (msg4)
#************************************
#Agreeing with escapes
#************************************
msg5 = r"c:\rootdir\otherdir"
msg6 = "c:\\rootdir\\otherdir"
#************************************
#Comparing strings
#************************************
if msg5 == msg6:
print ("yes they are same")
else:
print("No, they are not the same")
print (msg5)
print (msg6)
#************************************
#Finding strings using "in" key word
#************************************
available1 = r"c:\root" in msg5
available2 = "c:\root" in msg5
print (available1)
print (available2)
#************************************
#Subscripting
#************************************
msg5 = "0123456789"
msg7 = msg5[:4] #Get 0 to 3
print (msg7)
#Notice how it gets the 5th character
#as the array is 0 based
msg8 = msg5[4:8] #4,5,6,7
print (msg8)
c7th = msg5[6]
print (c7th) #You will get 6
#************************************
#Length
# len() is what is called a Built in function
#************************************
print (len(msg5)) #should be 10
#************************************
#Cases: Title, Upper, Lower
#************************************
msg9 = "how ABOUT This sEntence"
print (msg9.title()) #How About This Sentence
satya - 8/25/2019, 1:55:28 PM
OO Nature of Python
Python is an object oriented programming language.
Almost everything in Python is an object, with its properties and methods.
satya - 8/25/2019, 2:08:38 PM
Functions in Python
They are first class
don't need to be in an object
default args supported
indented for completion
types are inferred and not specified
arguments can be in sequence or named
satya - 8/25/2019, 2:09:06 PM
Example
def my_function(food):
for x in food:
print(x)
fruits = ["apple", "banana", "cherry"]
my_function(fruits)
satya - 8/25/2019, 2:09:53 PM
Arbitrary arguments with a * in python
Arbitrary arguments with a * in python
satya - 8/25/2019, 2:15:05 PM
Python collections
Lists: [a,b,c] #Changeable
Tuples: (a,b,c) #Unchangeable
Sets: {a,b,c} #unordered, unindexed
Dictionaries: {a:b, c:d} #uordered, changeable, indexed
satya - 8/28/2019, 2:36:31 PM
How to reference a self object in python?
How to reference a self object in python?
satya - 8/28/2019, 2:53:35 PM
Understand the following python modules
satya - 9/7/2019, 5:57:53 PM
Here is an example of a function definition
#***********************************
#Function: printCollected
#***********************************
def printBeginMsg(msg):
print ("*****************************")
print ("*" + msg)
print ("*****************************")
def printEndMsg(msg):
print ("*****************************")
print ("*" + msg)
print ("*****************************")
def printCollected(msg, collectedRDD):
printBeginMsg(msg)
for item in collectedRDD:
print (item)
printEndMsg(msg)
#***********************************
#End of function
#***********************************
satya - 9/11/2019, 10:46:25 AM
How do I know the type of an object in python?
How do I know the type of an object in python?
satya - 9/11/2019, 10:49:12 AM
Examples
#Print type
print (type(someobjectRef))
#examine type
class T
isinstance(object, T)
satya - 9/11/2019, 11:13:49 AM
Better approach to printing strings
# you cannot do this
print("blah " + type(o))
# what you have to do
type1 = type(o)
type2 = type(o)
print("blah type1 {} type2 {} etc ".format(type1,type2))
satya - 9/22/2019, 1:49:01 PM
The raw strings for mentioning directories on windows
# r before a string stands for raw
# This gives an error however
dataDirectory = r'C:\satya\data\code\pyspark\'
# SyntaxError: EOL while scanning string literal
#this is ok
dataDirectory = r'C:\satya\data\code\pyspark'
satya - 9/22/2019, 1:49:55 PM
Online Book: This python text book is a good online quick reference
Online Book: This python text book is a good online quick reference
satya - 9/22/2019, 2:02:34 PM
Some code
#Some constants
dataDirectory = r'C:\satya\data\code\pyspark'
processedDir = dataDirectory + r"\data\mp_scada_data\processed"
inputWuActualWeatherForAWeek = dataDirectory + r"\data\mp_scada_data\wu_actual\All Data.csv"
def printVar(msg, var1):
formatMsg = msg + ": {}"
finalMsg = formatMsg.format(var1)
print(finalMsg)
printVar ("Data directory", dataDirectory)
printVar ("inputWuActualWeatherForAWeek",inputWuActualWeatherForAWeek)
#You can also do this instead
print ("Data directory: ", dataDirectory)
print ("inputWuActualWeatherForAWeek: ",inputWuActualWeatherForAWeek)
satya - 9/22/2019, 2:29:26 PM
List comprehensions in python
List comprehensions in python
satya - 9/22/2019, 2:34:27 PM
Notice the : around if and else statements
if len(mylist) < 10:
print (mylist)
else:
print ("List too big")
satya - 9/22/2019, 2:59:00 PM
Understand enumerate in python comprehensions
Understand enumerate in python comprehensions
satya - 9/22/2019, 2:59:29 PM
Look at this code
mylist = [line for idx, line in enumerate(inputFileObj) if idx < 5]
print ("*********")
print (len(mylist))
print (mylist[0])
if len(mylist) < 10:
print (mylist)
else:
print ("List too big")
print (len(mylist))
satya - 9/23/2019, 10:40:03 AM
True and False
true: "dddd"
false: "" //empty
//All false because they are empty
//if not then true
[]
{}
()
false: 0
true: 1
false: None
satya - 9/23/2019, 10:42:35 AM
whats up with : in python
Whats up with :
Starts a block of code
In #python
Examples:
If ... :
for ... :
:)
satya - 9/23/2019, 10:43:25 AM
Optional parenthesis
if (x > y)
is ok to say
if x > y
satya - 9/23/2019, 10:45:24 AM
Apparently this is ok
if (a == 1 and
b == 2 and
c == blah ) :
other stuff
where brackets are extending a line. there may be nuances as brackets can define a tuple as well!!
satya - 9/23/2019, 10:49:06 AM
Unpacking lists with names
#unpack a list or tuple etc
a,b,c,d,e = "hello"
then
a = h
b = e
c= l
d = l
e = o
satya - 9/23/2019, 1:57:38 PM
working with Date time in python
working with Date time in python
satya - 9/23/2019, 2:21:40 PM
This seem to work
from datetime import datetime
someInstance = datetime.fromisoformat("2019-09-01 00:05:00.000")
print(someInstance)
#prints: 2019-09-01 00:05:00
satya - 9/23/2019, 2:21:51 PM
You can also do this
from datetime import datetime
someInstance = datetime.fromisoformat(
"2019-09-01 00:05:00.000"
)
print(someInstance)
satya - 9/23/2019, 2:23:22 PM
And this
from datetime import datetime
someInstance = datetime.fromisoformat(
"2019-09-01 00:05:00.000"
)
print(someInstance)
print(someInstance.year)
print(someInstance.month)
print(someInstance.day)
satya - 9/23/2019, 2:37:30 PM
string formatting integers in python
string formatting integers in python
satya - 9/23/2019, 2:48:22 PM
See this code
from datetime import datetime
sampleTimeInstanceString = "2019-09-01 00:05:00.000"
someInstance = datetime.fromisoformat(
sampleTimeInstanceString
)
print(someInstance)
print(someInstance.year)
print(someInstance.month)
print(someInstance.day)
def getFilename(datetimeInstanceString):
fileBasename = "somebase-filename-for-"
#i - instance
i = datetime.fromisoformat(datetimeInstanceString)
#see the formating options for padding
#integers with 0s
filePostFix = (
"{}{:02d}{:02d}{:02d}"
.format(i.year, i.month,i.day, i.hour)
)
return fileBasename + filePostFix + ".csv"
print (getFilename(sampleTimeInstanceString))
# will pring the following
# somebase-filename-for-2019090100.csv
satya - 9/23/2019, 2:49:02 PM
Specifically
filePostFix = (
"{}{:02d}{:02d}{:02d}"
.format(i.year, i.month,i.day, i.hour)
)
satya - 9/25/2019, 9:05:48 AM
Working with directories in python
Working with directories in python
satya - 9/25/2019, 9:06:34 AM
Here is os.path in the standard library of python
satya - 9/25/2019, 9:06:51 AM
There doesn't seem to be an explicit structure like a "directory"
There doesn't seem to be an explicit structure like a "directory"
satya - 9/25/2019, 9:09:58 AM
File and directory access in python docs
satya - 9/25/2019, 9:15:31 AM
working with files and directories in python
satya - 9/25/2019, 9:32:16 AM
How to raise exceptions in Python
How to raise exceptions in Python
satya - 9/25/2019, 9:32:30 AM
Here is the raise keyword in python lang reference
satya - 9/25/2019, 9:53:58 AM
class documentation in lang reference
satya - 9/25/2019, 9:59:41 AM
calling other methods for __init__ in python
calling other methods for __init__ in python
satya - 9/25/2019, 10:02:19 AM
Here is the way
#calling form init
#notice "self."
self.f1()
#define it
def f1(self):
satya - 9/26/2019, 10:42:11 AM
How to delete files from a folder
How to delete files from a folder
satya - 9/26/2019, 10:42:34 AM
How can I delete the contents of a local folder in Python?
How can I delete the contents of a local folder in Python?
Search for: How can I delete the contents of a local folder in Python?
satya - 9/26/2019, 10:42:52 AM
at SOF: How can I delete the contents of a local folder in Python?
at SOF: How can I delete the contents of a local folder in Python?
satya - 9/26/2019, 10:51:30 AM
The link above is a good source for
os
shutil
os.path
os.listdir
shutil.rmtree
os.unlink
glob.glob
os.remove(file)
satya - 9/26/2019, 11:06:34 AM
Here is an example
import os
from os import path
import shutil
def _emptyFilesetDirectory(self, fileSetDirectory):
for item in os.listdir(fileSetDirectory):
#item is just a name, not the full path
itemFullpath = path.join(fileSetDirectory,item)
if path.isfile(itemFullpath):
os.remove(itemFullpath)
continue
if path.isdir(itemFullpath):
shutil.rmtree(itemFullpath)
continue
satya - 10/14/2019, 6:33:19 PM
How to put imports on multiple lines in python
How to put imports on multiple lines in python
satya - 10/14/2019, 6:34:06 PM
You can do one of this
from SimpleXMLRPCServer import SimpleXMLRPCServer, \
SimpleXMLRPCRequestHandler, \
CGIXMLRPCRequestHandler, \
resolve_dotted_attribute
Or
from SimpleXMLRPCServer import (SimpleXMLRPCServer,
SimpleXMLRPCRequestHandler,
CGIXMLRPCRequestHandler,
resolve_dotted_attribute)
satya - 10/14/2019, 6:34:27 PM
These are called multi-line imports in python
These are called multi-line imports in python
satya - 10/14/2019, 6:34:37 PM
Multi-line imports are documented here
satya - 10/14/2019, 6:35:51 PM
Here is a combination
from outputfilesetdirectory import (getTheHour,
FileSetDirectory,
getFilenameByHour)
satya - 10/14/2019, 6:37:42 PM
Or this will work too
from outputfilesetdirectory import (
getTheHour,
FileSetDirectory,
getFilenameByHour
)
satya - 10/14/2019, 7:28:17 PM
python partially assign variables from a list
python partially assign variables from a list
satya - 10/14/2019, 7:28:58 PM
Partial list unpack in Python: SOF
satya - 10/14/2019, 7:35:02 PM
* operator multiple assignment syntax in python 3
* operator multiple assignment syntax in python 3
Search for: * operator multiple assignment syntax in python 3
satya - 10/14/2019, 7:35:25 PM
Multiple assignment and tuple unpacking improve Python code readability: article
Multiple assignment and tuple unpacking improve Python code readability: article
satya - 10/14/2019, 7:41:34 PM
I can do this using partial field assignments
def getFirstField(aRowWithCommaSeparatedFields):
#Get some type help by saying it is a string
line : str = aRowWithCommaSeparatedFields
#line is a set of fields
#First field is an iso time format
#2019-09-01 00:05:00.000
#Many other comma separated fields
#Using type help you can discover .partition method
timeField, *rest = line2.partition(",")
print(timeField)
print(line)
satya - 10/18/2019, 1:12:22 PM
with statement to open and write to a file in python
with statement to open and write to a file in python
Search for: with statement to open and write to a file in python
satya - 10/18/2019, 1:17:11 PM
with statement in python
with statement in python
satya - 10/18/2019, 1:37:21 PM
Brief summary
with someObject as x:
suite of statements
that use x
could be equivalent to
x = someobject.enter(self)
execute suite
someobject.exit(self, exc_type, exc_value, traceback)
satya - 10/18/2019, 1:40:35 PM
Some explanation
1. if enter succeeds, it is guaranteed that exit is called.
2. if no exception, then None objects are passed to exit
3. if exception, then those exception objects are passed in
4. exit can return true to stop the propagation of exception otherwise the exception is passed on
satya - 10/18/2019, 1:41:57 PM
So with a file object you can do
with open(filename, filemode) as myfile:
myfile.write(...)
The myfile object will be guaranteed to be closed irrespective of the exception or success
satya - 10/18/2019, 1:43:01 PM
Working with context managers in python
Working with context managers in python
satya - 10/18/2019, 1:46:08 PM
Context managers implement the idea of
//Ensure this
Do some thing before
//then do this
Do code
//Ensure this even with failures
Do something after
satya - 10/18/2019, 1:58:58 PM
Context Managers are documented in GitHub book PythonTips book by Yasoob
Context Managers are documented in GitHub book PythonTips book by Yasoob
satya - 10/18/2019, 2:55:29 PM
Idea of using Enum for file modes open() in python is discussed here
Idea of using Enum for file modes open() in python is discussed here
satya - 10/18/2019, 2:55:59 PM
Using Enum for file modes open() in python
Using Enum for file modes open() in python
satya - 10/18/2019, 3:18:23 PM
Perhaps I can do this instead
class FileMode:
#Some popular modes
#Reads
r = "r"
read = r
#writes
w = "w"
write = w #overwrite
#append
a = "a"
append = a
#Attach this where needed
binary = "b"
satya - 10/18/2019, 3:21:02 PM
That will jog the memory when I do
#Notice the FileMode. for intellisense
with open(filepath, FileMode.w) as file:
for line in actualWeatherByTheHourList:
file.write(line)
satya - 7/3/2020, 9:51:29 AM
Syntax of a list comprehension
[x for x in list if x % 2 == 0]
or
[f2(x) for x in list if f(x) == true]
same as
select x // before for
from table //for part
where f(x) == true //if part
satya - 7/12/2020, 8:55:43 PM
type hints in python
type hints in python
satya - 7/12/2020, 8:57:44 PM
Example
def greeting(name: str) -> str:
return 'Hello ' + name
satya - 9/18/2022, 3:10:49 PM
Multi-line comment in python
Multi-line comment in python
satya - 9/18/2022, 3:12:24 PM
PYthon does not have multi-line comments.
PYthon does not have multi-line comments.
satya - 1/20/2024, 1:02:49 PM
For intellisense in python in vscode do this
satya - 1/20/2024, 1:03:40 PM
Now the following is very useful
#In your function give a type hint as below
def understandResponse(response: requests.Response):
response.
After the "." tupe ctrl-space, and you will ONLY those methods specific to that object.
satya - 1/20/2024, 3:05:41 PM
This is possible in Python: Cascading imports
lib1 imports lib2
App import lib1 and also lib2
In other words
lib1 -> lib2
App -> lib1, lib2
satya - 1/20/2024, 3:05:57 PM
Cascading imports are common in Python
Cascading imports are common in Python
satya - 1/21/2024, 8:07:32 PM
Useful in libs to execute them
def localTest():
print ("Starting local test")
print ("End local test")
if __name__ == '__main__':
localTest()
satya - 2/1/2024, 12:39:24 PM
try catch
try:
# Code that may raise an exception
num = int(input("Enter a number: "))
result = 10 / num # This line may raise a ZeroDivisionError if num is 0
print("Result:", result)
except ZeroDivisionError:
# Handle the specific exception (ZeroDivisionError)
print("Error: Cannot divide by zero")
except ValueError:
# Handle another specific exception (ValueError)
print("Error: Invalid input. Please enter a valid number")
except Exception as e:
# Handle any other exception not caught by the specific except blocks
print(f"An error occurred: {e}")
else:
# Optional else block that executes if no exceptions occurred
print("No exceptions occurred")
finally:
# Optional finally block that always executes, whether there's an exception or not
print("Execution finished")
satya - 2/2/2024, 4:09:33 PM
enum code
class Color(Enum):
RED = 1
GREEN = 2
BLUE = 3
# then
if (Color.Red == 1) etc.
satya - 2/2/2024, 4:10:22 PM
sub strings
main_string = "Hello, world!"
sub_string = "world"
if sub_string in main_string:
print(f"'{sub_string}' is present in '{main_string}'")
else:
print(f"'{sub_string}' is not present in '{main_string}'")