Error handling in ETL
satya - 4/12/2020, 11:57:04 AM
The usual classification of errors
1. info
2. warning
3. error
4. secure
satya - 4/12/2020, 12:25:28 PM
How about futher
1. debug
2, trace
3. fatal
satya - 4/12/2020, 12:25:59 PM
Type of exceptions or errors
1. record level
2. file level
3. global level
satya - 4/12/2020, 12:27:27 PM
Further for each level are there kinds that needs a different treatment
1. File level, but needs to stop other files
3. File level, but actions differ what that error means, for example don't move the file and consider rerunning it, etc.
satya - 4/12/2020, 12:27:38 PM
how to send alerts based on these errors
how to send alerts based on these errors
satya - 4/12/2020, 12:28:50 PM
Approach for
1. archive processed files
2. when to remove files for backup
3. how long to retain
4. log files retention and removal
satya - 4/12/2020, 12:31:39 PM
How to reduce the amount of logging?
How to reduce the amount of logging?
satya - 4/12/2020, 12:54:46 PM
Order of validation
1. validateParams
2. copy to local variables
3. validate local variables
satya - 4/12/2020, 12:56:55 PM
More on order of validation
1. validateParams
2. validate necessary environment variables
3. copy to local variables from params
4. copy from env variables to local
3. validate local variables
satya - 4/12/2020, 1:08:22 PM
Simulation can help a lot...to debug and test
Simulation can help a lot...to debug and test
satya - 4/12/2020, 11:00:38 PM
flowlog: just like info, error, and warning
1. A better name desirable
2. flowlog is a log statement which when printed and nothing else shows the flow of an EtL without overwhelming the logs.