Data factory basics
satya - 10/26/2019, 1:08:30 PM
Copy data from SFTP server using Azure Data Factory
satya - 10/26/2019, 1:26:26 PM
Mapping data flows availability: 2019
satya - 10/26/2019, 2:43:00 PM
SFTP linked service, limiting the number of files copied
SFTP linked service, limiting the number of files copied
Search for: SFTP linked service, limiting the number of files copied
satya - 10/26/2019, 3:15:52 PM
How to ask a question on Azure portal in their forums
satya - 10/26/2019, 3:16:24 PM
Here is my specific question around adf2 sftp connector
satya - 10/26/2019, 3:16:52 PM
Posted question link: How do I control the number of files copied by ADF v2 SFTP connector?
Posted question link: How do I control the number of files copied by ADF v2 SFTP connector?
satya - 10/26/2019, 3:52:16 PM
A document on incremental data copies
A document on incremental data copies
Something to read, and see, not sure if this addresses the question.
satya - 10/26/2019, 4:02:53 PM
I will need this later: Transform data in the cloud by using a Spark activity in Azure Data Factory
I will need this later: Transform data in the cloud by using a Spark activity in Azure Data Factory
satya - 10/27/2019, 9:23:54 AM
Here is how to understand templates in adf
satya - 10/27/2019, 12:32:24 PM
Can azure data factory save state information between runs?
Can azure data factory save state information between runs?
Search for: Can azure data factory save state information between runs?
satya - 10/27/2019, 12:35:27 PM
Article: USING AZURE DATA FACTORY V2 ACTIVITIES & DYNAMIC CONTENT TO DIRECT YOUR FILES
Article: USING AZURE DATA FACTORY V2 ACTIVITIES & DYNAMIC CONTENT TO DIRECT YOUR FILES
satya - 10/27/2019, 1:19:16 PM
Using python to create a data factory
satya - 10/27/2019, 1:41:00 PM
I have posted a question on debugging activities here
satya - 10/28/2019, 11:36:23 AM
not able to connect to sftp server from azure data factory v2
not able to connect to sftp server from azure data factory v2
Search for: not able to connect to sftp server from azure data factory v2
satya - 11/1/2019, 5:45:58 PM
Whats new in adf v2 (2017): a weblog
Whats new in adf v2 (2017): a weblog
Looks like a good high level overview of all the pieces of adf v2
satya - 11/1/2019, 6:02:58 PM
Poster a question on azure forum: Can I call an azure function from Lookup Activity to gather a dynamic set
satya - 11/2/2019, 9:30:16 AM
How do I navigate to the data factory screens?
1. Go to portal
2. Go to home
3. click on data factories icon
4. Go to author/monitor
satya - 11/2/2019, 10:16:30 AM
Common reasons why an ftp connector fails
1. sftp uses port 22
2. ftp uses port 21
3. Using a wrong connector ftp for sftp or sftp for ftp could result in an error
4. when using ftp, if the ftp server is not enabled for SSL. (In this case disable ssl assuming it is safe for your needs). If not debug the ftp server and fix the issue
satya - 11/2/2019, 11:09:06 AM
One reason GetMetadata may fail
1. You have to specify the output fields desired from the metadata of a data source
2. If a particular output field is not supported, you get an error
3. Remove that field from the outputs
satya - 11/2/2019, 11:13:04 AM
Linked services is documented here
satya - 11/2/2019, 11:45:07 AM
Annotations in general and on a linked service
1. These are tags
2. You can name any number of tags on any component including a linked service
satya - 11/2/2019, 11:45:38 AM
This video briefly touches on this aspect: tags and annotations
This video briefly touches on this aspect: tags and annotations
satya - 11/2/2019, 12:00:02 PM
Here is how an ftp linked service gets encoded
{
"name": "your-linked-servicename",
"type": "Microsoft.DataFactory/factories/linkedservices",
"properties": {
"description": "your-ftp-server-description",
"annotations": [
"ingest",
"another-annotation-name"
],
"type": "FtpServer",
"typeProperties": {
"host": "ftp-hostname",
"port": 21,
"enableSsl": false,
"enableServerCertificateValidation": false,
"authenticationType": "Basic",
"userName": "user-name",
"encryptedCredential": "some-letters"
}
}
}
satya - 11/2/2019, 12:04:44 PM
specify dynamic content in json format adf v2
specify dynamic content in json format adf v2
satya - 11/2/2019, 12:07:49 PM
Interesting diversion: Channel 9: https://channel9.msdn.com
satya - 11/3/2019, 10:48:20 AM
question on azure forum: dynamic content and linked service
satya - 11/3/2019, 10:55:09 AM
There is a video here: Parameterize connections to your data stores in Azure Data Factory
There is a video here: Parameterize connections to your data stores in Azure Data Factory
satya - 11/3/2019, 10:58:22 AM
Parameterizing linked services is documented here
satya - 11/3/2019, 11:01:46 AM
First document I have seen on expression language
satya - 11/3/2019, 11:33:11 AM
Only certain data stores support UI parameterization fo now
Azure SQL Database
Azure SQL Data Warehouse
SQL Server
Oracle
Cosmos DB
Amazon Redshift
MySQL
Azure Database for MySQL
satya - 11/3/2019, 11:33:34 AM
For the others, it says
For all other data stores, you can parameterize the linked service by selecting the Code icon on the Connections tab and using the JSON editor.
satya - 11/3/2019, 12:12:59 PM
Here is a connector with dynamic content
{
"name": "AzureSqlDatabase",
"properties": {
"type": "AzureSqlDatabase",
"typeProperties": {
"connectionString": {
"value": "Server=tcp:myserver.database.windows.net,1433;\
Database=@{linkedService().DBName};\
User ID=user;\
Password=fake; \
Trusted_Connection=False;\
Encrypt=True;\
Connection Timeout=30",
"type": "SecureString"
}
},
"connectVia": null,
"parameters": {
"DBName": {
"type": "String"
}
}
}
}
satya - 11/3/2019, 12:16:25 PM
Passing parameters between activities and pipelines: A PDF
satya - 11/3/2019, 12:31:15 PM
Some prefixes and naming conventions
pl_ //pipeline
ds_ //data set
ac_ //activity
ls_ //linkedservice
satya - 11/3/2019, 12:39:18 PM
Too sad, that is not a particularly good document!!
Too sad, that is not a particularly good document!!
satya - 11/3/2019, 12:39:45 PM
Documentation: Visual authoring in Azure Data Factory
satya - 11/3/2019, 1:27:39 PM
I have a question posted here for the advanced tab
satya - 11/4/2019, 10:07:01 AM
I have posted some questions at youtube
satya - 11/4/2019, 10:26:40 AM
I have posted some questions to azure linked in group as well
I have posted some questions to azure linked in group as well
satya - 11/4/2019, 12:02:44 PM
How to use filter activity in adf v2
How to use filter activity in adf v2
satya - 11/4/2019, 12:10:16 PM
Filter activity is documented here at MS
satya - 11/5/2019, 11:43:18 AM
ftp linked service (connector) is documented here
satya - 11/5/2019, 11:46:21 AM
Here is what it says about the folder path and file path :(
The path to folder. If you want to use wildcard to filter folder, skip this setting and specify in activity source settings.
satya - 11/5/2019, 11:52:22 AM
Can you use wild cards for Get Metadata items?
Can you use wild cards for Get Metadata items?
satya - 11/5/2019, 1:48:24 PM
Output from a Get Metadata
{
"itemName": "wrf_wind.2019092312",
"childItems": [
{
"name": "output_stn01.txt",
"type": "File"
},
{
"name": "output_stn02.txt",
"type": "File"
},
{
"name": "output_stn03.txt",
"type": "File"
},
{
"name": "output_stn04.txt",
"type": "File"
},
{
"name": "output_stn05.txt",
"type": "File"
},
{
"name": "output_stn06.txt",
"type": "File"
},
{
"name": "output_stn07.txt",
"type": "File"
},
{
"name": "output_stn08.txt",
"type": "File"
},
{
"name": "output_stn09.txt",
"type": "File"
},
{
"name": "output_stn11.txt",
"type": "File"
},
{
"name": "output_stn16.txt",
"type": "File"
},
{
"name": "output_stn17.txt",
"type": "File"
},
{
"name": "output_stn18.txt",
"type": "File"
},
{
"name": "output_stn19.txt",
"type": "File"
},
{
"name": "readme.txt",
"type": "File"
},
{
"name": "zlevs_output_d02.nc",
"type": "File"
},
{
"name": "zlevs_output_d03.nc",
"type": "File"
},
{
"name": "zlevs_output_d04.nc",
"type": "File"
}
],
"effectiveIntegrationRuntime": "DefaultIntegrationRuntime (Central India)",
"executionDuration": 0,
"durationInQueue": {
"integrationRuntimeQueue": 0
}
}
satya - 11/5/2019, 1:48:42 PM
Appears to be a good article on filter activity
satya - 11/5/2019, 1:53:31 PM
Here is the filtered output
{
"ActivityRunId": "9c2a1a2c-cdf1-445b-acc4-186a984aafd8",
"Status": "Succeeded",
"Error": {
"Message": "",
"ErrorCode": ""
},
"Output": {
"ItemsCount": 18,
"FilteredItemsCount": 15,
"Value": [
{
"name": "output_stn01.txt",
"type": "File"
},
{
"name": "output_stn02.txt",
"type": "File"
},
{
"name": "output_stn03.txt",
"type": "File"
},
{
"name": "output_stn04.txt",
"type": "File"
},
{
"name": "output_stn05.txt",
"type": "File"
},
{
"name": "output_stn06.txt",
"type": "File"
},
{
"name": "output_stn07.txt",
"type": "File"
},
{
"name": "output_stn08.txt",
"type": "File"
},
{
"name": "output_stn09.txt",
"type": "File"
},
{
"name": "output_stn11.txt",
"type": "File"
},
{
"name": "output_stn16.txt",
"type": "File"
},
{
"name": "output_stn17.txt",
"type": "File"
},
{
"name": "output_stn18.txt",
"type": "File"
},
{
"name": "output_stn19.txt",
"type": "File"
},
{
"name": "readme.txt",
"type": "File"
}
]
}
}
satya - 11/5/2019, 1:55:33 PM
Here is the configuration for the filtered activity
//items to filter
items =
@activity('Get Metadata2').output.childItems
condition =
@endswith(item().name, '.txt')
satya - 11/5/2019, 1:57:26 PM
Here is the source code
{
"name": "TextFileFilter",
"description": "Text file Filter",
"type": "Filter",
"dependsOn": [
{
"activity": "Get Metadata2",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"items": {
"value": "@activity('Get Metadata2').output.childItems",
"type": "Expression"
},
"condition": {
"value": "@endswith(item().name, '.txt')",
"type": "Expression"
}
}
}
satya - 11/7/2019, 8:55:45 AM
Folder and file property for Copy Activity in ADF v2 azure data factory
Folder and file property for Copy Activity in ADF v2 azure data factory
Search for: Folder and file property for Copy Activity in ADF v2 azure data factory
satya - 11/7/2019, 12:06:09 PM
Azure custom activity is documented here
satya - 11/7/2019, 12:19:37 PM
Here is an azure function activity
satya - 11/7/2019, 12:47:30 PM
Azure functions core tools is here
satya - 11/14/2019, 10:52:25 AM
A complex example of file copy behavior
satya - 11/14/2019, 11:03:31 AM
Parameter passing to datasets in adf v2
Parameter passing to datasets in adf v2
satya - 11/14/2019, 11:04:01 AM
Parameter passing to datasets in adf v2