Stack Overflow
Stack Overflow

Questions tagged [azure-synapse-pipeline]

21 questions
1
vote
1 answer

Azure Synapse Pipeline Get Metadata Last Modified Date on file instead of on folder

I am trying to get the Last Modified date on the files (CSV linked service) in my repository using Get Metadata activity. Field List However, I am getting the same date/time for all of the files... the LastModified date on the folder instead of the…
CFCJB
  • 11
  • 4
1
vote
1 answer

Azure Synapse Pipeline How to execute each file as sql script?

Is it possible to use Azure Synapse Pipeline to execute a set of files as SQL scripts ? I have a set of files each of which contains the SQL to create say a stored procedure in my Azure Synapse Dedicated SQL pool. I have saved these SQL files in an…
Scott Stensland
  • 26,870
  • 12
  • 93
  • 104
0
votes
1 answer

Fetching data from On-Premise Sql Server to Azure Synapse Notebook using Linked Service

I have made a self hosted integration runtime and created a linked service to sql server but having problem in fetching data in synapse notebook using pyspark code. I want pyspark code for Fetching data from On-Premise Sql Server to Azure Synapse…
Akhaya Chandan Mishra
  • 78
  • 6
0
votes
1 answer

Add cross apply in an Azure Data Factory Data Flow

How do I simulate the cross apply in an Azure Data Factory Data flow? SELECT wo.OrderId, max(wo.OrderVersionId) as OrderVersionId, max(tss.FromDate) as TSSFromDate, tss.ToDate as TSSToDate FROM dayshift.OrderMain wo CROSS APPLY ( …
datainstints
  • 5
  • 2
0
votes
2 answers

Is it possible to create a date array variable dynamically in Azure Synapse Analytics?

In Azure Synapse Analytics, I would like to use the pipeline function to set a variable of type array for the date from 12 months ago to current date. The date in the variable is the first day of each month. For example, a variable to be created…
CuteeeeRabbit
  • 27
  • 7
0
votes
0 answers

String to Array in pipeline builder expression

My Synapse notebook returns an array, [1, 2] I would like to Iterate through each element using foreach activity, How do I convert it to a array from string, so foreach activity runs for each element. to keep it simple, I have created pipeline as…
subro
  • 1,167
  • 4
  • 20
  • 32
0
votes
1 answer

How to programmatically retrieve errors from an azure synapse pipeline?

How can I programmatically retrieve error messages from each run of an azure synapse pipeline ? Obviously if this was a normal binary I could redirect stderr and/or stdout by executing the binary from a shell wrapper, however its not that simple…
Scott Stensland
  • 26,870
  • 12
  • 93
  • 104
0
votes
1 answer

How to filter data from a MongoDB source to match _id(s) in a CosmosDB container using Azure Synapse Analytics?

I'm currently working with MongoDB and Azure CosmosDB for a project and I need to filter and copy data from one to the other. The source data is a MongoDB collection named "Purchase", where each record has an 'AccountId'. I have a container in…
Alireza Ghaffari
  • 1,004
  • 3
  • 11
  • 24
0
votes
2 answers

Sqlpackage.exe stucks on initialization

Am using sqlpackage.exe for synapse deployment, version 162.0.52.1. During the initialization, it stucks at this step: Does anybody know, how to solve it?
R. Maier
  • 340
  • 2
  • 13
0
votes
2 answers

Add Isnull join condition to Azure data factory data flow

I have this sql script : I need to add the "ON UpdatedWOV.Id IS NULL AND CTH.Id = PT.TranId" to a dataflow. My challenge is specifically how to add "ON UpdatedWOV IS NULL" please note that there are other join conditions that doesn't require the…
datainstints
  • 5
  • 2
0
votes
2 answers

Synapse pipeline - Databricks Activity Not Supported

I am facing the below error when a Synapse pipeline with Databricks notebook activity is used - Error code: OK Inner error code: BadRequest Message: Databricks Activity Not Supported. This seems to be a bug in the Synapse Studio as everything was…
Sanket Kelkar
  • 129
  • 2
  • 9
0
votes
1 answer

string_agg substitute used in Derive transformation in Azure data flow expression language

How do I convert this sql code "left(string_agg(zationClient, '; ') within group (order by zationClient) , 100)" as a Transformation in data factory specifically in azure data flow expression language? I am expecting an output like this with…
datainstints
  • 5
  • 2
0
votes
1 answer

Can I use parameterization to copy only specific columns from a number of parquet files

I have a bunch of Salesforce objects being copied into an Azure Data Lake Storage Gen 2 as parquet files. I want to copy these into an Azure Synapse/Dedicated SQL Pool as tables, but I don't want every column, I only need a fraction of them. I know…
JBSilverAge
  • 1
0
votes
2 answers

Why is "tokenlibrary" incompatible with a REST linked service

Within my Synapse notebook, not all of the Synapse "linked services" are available to use. I can connect to linked services for KeyVault and SQL and ADLS, but for some reason I cannot connect to a linked service for a REST API. Why is this…
David Beavon
  • 1,141
  • 9
  • 16
0
votes
1 answer

Auto Detect Data Types with Azure Synapse Analytics Copy Data (Import)

I have an Azure Data Lake Storage Gen2 account linked to an Azure Synapse Analytics (ASA) Workspace. The storage account has TIOBE Test data that looks like…
Shawn Eary
  • 684
  • 2
  • 7
  • 21
1
2