satya - 10/26/2019, 12:25:14 PM

Data factory documentation

satya - 10/26/2019, 12:28:31 PM

Overview of Datafactory

satya - 10/26/2019, 12:30:04 PM

Pricing is here

satya - 10/26/2019, 1:08:30 PM

Copy data from SFTP server using Azure Data Factory

satya - 10/26/2019, 1:26:26 PM

Mapping data flows availability: 2019

satya - 10/26/2019, 1:30:36 PM

Copy data tool is documented here

satya - 10/26/2019, 1:32:50 PM

Overview of copydata tool is here

satya - 10/26/2019, 2:43:00 PM

SFTP linked service, limiting the number of files copied

Search for: SFTP linked service, limiting the number of files copied

satya - 10/26/2019, 3:15:52 PM

How to ask a question on Azure portal in their forums

satya - 10/26/2019, 3:16:24 PM

Here is my specific question around adf2 sftp connector

satya - 10/26/2019, 3:16:52 PM

Posted question link: How do I control the number of files copied by ADF v2 SFTP connector?

satya - 10/26/2019, 3:52:16 PM

A document on incremental data copies

Something to read, and see, not sure if this addresses the question.

satya - 10/26/2019, 4:02:53 PM

I will need this later: Transform data in the cloud by using a Spark activity in Azure Data Factory

satya - 10/27/2019, 9:23:54 AM

Here is how to understand templates in adf

satya - 10/27/2019, 9:24:12 AM

A template for moving files

satya - 10/27/2019, 12:32:24 PM

Can azure data factory save state information between runs?

Search for: Can azure data factory save state information between runs?

satya - 10/27/2019, 12:35:27 PM

Article: USING AZURE DATA FACTORY V2 ACTIVITIES & DYNAMIC CONTENT TO DIRECT YOUR FILES

satya - 10/27/2019, 1:19:16 PM

Using python to create a data factory

satya - 10/27/2019, 1:41:00 PM

I have posted a question on debugging activities here

satya - 10/28/2019, 11:35:58 AM

outbound connections in azure

satya - 10/28/2019, 11:36:23 AM

not able to connect to sftp server from azure data factory v2

Search for: not able to connect to sftp server from azure data factory v2

satya - 11/1/2019, 5:45:58 PM

Whats new in adf v2 (2017): a weblog

Looks like a good high level overview of all the pieces of adf v2

satya - 11/1/2019, 6:02:58 PM

Poster a question on azure forum: Can I call an azure function from Lookup Activity to gather a dynamic set

satya - 11/2/2019, 9:30:16 AM

How do I navigate to the data factory screens?

1. Go to portal

2. Go to home

3. click on data factories icon

4. Go to author/monitor

satya - 11/2/2019, 10:16:30 AM

Common reasons why an ftp connector fails

1. sftp uses port 22

2. ftp uses port 21

3. Using a wrong connector ftp for sftp or sftp for ftp could result in an error

4. when using ftp, if the ftp server is not enabled for SSL. (In this case disable ssl assuming it is safe for your needs). If not debug the ftp server and fix the issue

satya - 11/2/2019, 11:09:06 AM

One reason GetMetadata may fail

1. You have to specify the output fields desired from the metadata of a data source

2. If a particular output field is not supported, you get an error

3. Remove that field from the outputs

satya - 11/2/2019, 11:13:04 AM

Linked services is documented here

satya - 11/2/2019, 11:45:07 AM

Annotations in general and on a linked service

1. These are tags

2. You can name any number of tags on any component including a linked service

satya - 11/2/2019, 11:45:38 AM

This video briefly touches on this aspect: tags and annotations

satya - 11/2/2019, 12:00:02 PM

Here is how an ftp linked service gets encoded


{
    "name": "your-linked-servicename",
    "type": "Microsoft.DataFactory/factories/linkedservices",
    "properties": {
        "description": "your-ftp-server-description",
        "annotations": [
            "ingest",
            "another-annotation-name"
        ],
        "type": "FtpServer",
        "typeProperties": {
            "host": "ftp-hostname",
            "port": 21,
            "enableSsl": false,
            "enableServerCertificateValidation": false,
            "authenticationType": "Basic",
            "userName": "user-name",
            "encryptedCredential": "some-letters"
        }
    }
}

satya - 11/2/2019, 12:04:44 PM

specify dynamic content in json format adf v2

Search for: specify dynamic content in json format adf v2

satya - 11/2/2019, 12:07:49 PM

Interesting diversion: Channel 9: https://channel9.msdn.com

satya - 11/2/2019, 12:09:30 PM

Azure friday videos

satya - 11/3/2019, 10:48:20 AM

question on azure forum: dynamic content and linked service

satya - 11/3/2019, 10:55:09 AM

There is a video here: Parameterize connections to your data stores in Azure Data Factory

satya - 11/3/2019, 10:58:22 AM

Parameterizing linked services is documented here

satya - 11/3/2019, 11:01:46 AM

First document I have seen on expression language

satya - 11/3/2019, 11:33:11 AM

Only certain data stores support UI parameterization fo now


Azure SQL Database
Azure SQL Data Warehouse
SQL Server
Oracle
Cosmos DB
Amazon Redshift
MySQL
Azure Database for MySQL

satya - 11/3/2019, 11:33:34 AM

For the others, it says

For all other data stores, you can parameterize the linked service by selecting the Code icon on the Connections tab and using the JSON editor.

satya - 11/3/2019, 12:12:59 PM

Here is a connector with dynamic content


{
   "name": "AzureSqlDatabase",
   "properties": {
      "type": "AzureSqlDatabase",
      "typeProperties": {
         "connectionString": {
            "value": "Server=tcp:myserver.database.windows.net,1433;\ 
             Database=@{linkedService().DBName};\ 
             User ID=user;\ 
             Password=fake; \ 
             Trusted_Connection=False;\ 
             Encrypt=True;\ 
             Connection Timeout=30",
            "type": "SecureString"
         }
      },
      "connectVia": null,
      "parameters": {
         "DBName": {
            "type": "String"
         }
      }
   }
}

satya - 11/3/2019, 12:16:25 PM

Passing parameters between activities and pipelines: A PDF

satya - 11/3/2019, 12:31:15 PM

Some prefixes and naming conventions


pl_  //pipeline
ds_  //data set
ac_  //activity
ls_  //linkedservice

satya - 11/3/2019, 12:39:18 PM

Too sad, that is not a particularly good document!!

satya - 11/3/2019, 12:39:45 PM

Documentation: Visual authoring in Azure Data Factory

satya - 11/3/2019, 1:27:39 PM

I have a question posted here for the advanced tab

satya - 11/4/2019, 10:07:01 AM

I have posted some questions at youtube

satya - 11/4/2019, 10:26:40 AM

I have posted some questions to azure linked in group as well

satya - 11/4/2019, 12:02:44 PM

How to use filter activity in adf v2

Search for: How to use filter activity in adf v2

satya - 11/4/2019, 12:10:16 PM

Filter activity is documented here at MS

satya - 11/5/2019, 11:43:18 AM

ftp linked service (connector) is documented here

satya - 11/5/2019, 11:46:21 AM

Here is what it says about the folder path and file path :(

The path to folder. If you want to use wildcard to filter folder, skip this setting and specify in activity source settings.

satya - 11/5/2019, 11:52:22 AM

Can you use wild cards for Get Metadata items?

Search for: Can you use wild cards for Get Metadata items?

satya - 11/5/2019, 1:48:24 PM

Output from a Get Metadata


{
    "itemName": "wrf_wind.2019092312",
    "childItems": [
        {
            "name": "output_stn01.txt",
            "type": "File"
        },
        {
            "name": "output_stn02.txt",
            "type": "File"
        },
        {
            "name": "output_stn03.txt",
            "type": "File"
        },
        {
            "name": "output_stn04.txt",
            "type": "File"
        },
        {
            "name": "output_stn05.txt",
            "type": "File"
        },
        {
            "name": "output_stn06.txt",
            "type": "File"
        },
        {
            "name": "output_stn07.txt",
            "type": "File"
        },
        {
            "name": "output_stn08.txt",
            "type": "File"
        },
        {
            "name": "output_stn09.txt",
            "type": "File"
        },
        {
            "name": "output_stn11.txt",
            "type": "File"
        },
        {
            "name": "output_stn16.txt",
            "type": "File"
        },
        {
            "name": "output_stn17.txt",
            "type": "File"
        },
        {
            "name": "output_stn18.txt",
            "type": "File"
        },
        {
            "name": "output_stn19.txt",
            "type": "File"
        },
        {
            "name": "readme.txt",
            "type": "File"
        },
        {
            "name": "zlevs_output_d02.nc",
            "type": "File"
        },
        {
            "name": "zlevs_output_d03.nc",
            "type": "File"
        },
        {
            "name": "zlevs_output_d04.nc",
            "type": "File"
        }
    ],
    "effectiveIntegrationRuntime": "DefaultIntegrationRuntime (Central India)",
    "executionDuration": 0,
    "durationInQueue": {
        "integrationRuntimeQueue": 0
    }
}

satya - 11/5/2019, 1:48:42 PM

Appears to be a good article on filter activity

satya - 11/5/2019, 1:53:31 PM

Here is the filtered output


{
    "ActivityRunId": "9c2a1a2c-cdf1-445b-acc4-186a984aafd8",
    "Status": "Succeeded",
    "Error": {
        "Message": "",
        "ErrorCode": ""
    },
    "Output": {
        "ItemsCount": 18,
        "FilteredItemsCount": 15,
        "Value": [
            {
                "name": "output_stn01.txt",
                "type": "File"
            },
            {
                "name": "output_stn02.txt",
                "type": "File"
            },
            {
                "name": "output_stn03.txt",
                "type": "File"
            },
            {
                "name": "output_stn04.txt",
                "type": "File"
            },
            {
                "name": "output_stn05.txt",
                "type": "File"
            },
            {
                "name": "output_stn06.txt",
                "type": "File"
            },
            {
                "name": "output_stn07.txt",
                "type": "File"
            },
            {
                "name": "output_stn08.txt",
                "type": "File"
            },
            {
                "name": "output_stn09.txt",
                "type": "File"
            },
            {
                "name": "output_stn11.txt",
                "type": "File"
            },
            {
                "name": "output_stn16.txt",
                "type": "File"
            },
            {
                "name": "output_stn17.txt",
                "type": "File"
            },
            {
                "name": "output_stn18.txt",
                "type": "File"
            },
            {
                "name": "output_stn19.txt",
                "type": "File"
            },
            {
                "name": "readme.txt",
                "type": "File"
            }
        ]
    }
}

satya - 11/5/2019, 1:55:33 PM

Here is the configuration for the filtered activity


//items to filter
items = 
  @activity('Get Metadata2').output.childItems

condition =
  @endswith(item().name, '.txt')

satya - 11/5/2019, 1:57:26 PM

Here is the source code


{
    "name": "TextFileFilter",
    "description": "Text file Filter",
    "type": "Filter",
    "dependsOn": [
        {
            "activity": "Get Metadata2",
            "dependencyConditions": [
                "Succeeded"
            ]
        }
    ],
    "userProperties": [],
    "typeProperties": {
        "items": {
            "value": "@activity('Get Metadata2').output.childItems",
            "type": "Expression"
        },
        "condition": {
            "value": "@endswith(item().name, '.txt')",
            "type": "Expression"
        }
    }
}

satya - 11/7/2019, 8:53:19 AM

Copy activity is documented here

satya - 11/7/2019, 8:55:45 AM

Folder and file property for Copy Activity in ADF v2 azure data factory

Search for: Folder and file property for Copy Activity in ADF v2 azure data factory

satya - 11/7/2019, 12:06:09 PM

Azure custom activity is documented here

satya - 11/7/2019, 12:19:37 PM

Here is an azure function activity

satya - 11/7/2019, 12:31:00 PM

writing an azure python function

satya - 11/7/2019, 12:47:30 PM

Azure functions core tools is here

satya - 11/14/2019, 10:52:25 AM

A complex example of file copy behavior

satya - 11/14/2019, 11:03:31 AM

Parameter passing to datasets in adf v2

Search for: Parameter passing to datasets in adf v2

satya - 11/14/2019, 11:04:01 AM

Parameter passing to datasets in adf v2