Handling non UTF8 encoding in Logic App
Recently there was a requirement to generate a flat file in Azure Logic App and deliver to Azure File Share with ANSI encoding as targeting application could only process ANSI encoding file. Much of cloud services assume that a text payload will be some form of UTF (Unicode) encoding. Azure Logic App Assumes it is UTF-8. Such that when your text payload is in a different encoding, such as a page code based encoding, the non-basic Latin characters gets mangled. This is particularly common with Flat Files because they integrate with ancient systems that often were not written with Unicode support.
My approach to solving the problem was to create an Azure Function App that converts the encoding from UTF-8 to windows-1252 (or to any other encoding) and then stores the file content in Azure File storage.
This seems to be an easy fix but the main problem was that Azure Logic App did not like the output from Azure Function App and threw an exception as shown below
BadRequest. Http request failed as the content was not valid: ‘Unable to translate bytes [E4] at index 83 from specified code page to Unicode.’.
The solution would be to use Base 64 encoding. Base 64 encoding ensures that none of the services in Azure integration going to assume a UTF encoding. Once you convert any non-UTF flat file such (as windows-1252) to UTF-8, then base 64 decodes it safely and process it with flat-file decode.
Azure Function App to change the encoding
The following azure function app can be used to convert the encoding of the text in base64 encoding
using System;
using System.Net;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Newtonsoft.Json;
namespace PnJ.FunctionApp.ConvertEncoding
{
public static class ChangeBase64Encoding
{
[FunctionName("ChangeBase64Encoding")]
public static async Task<object> Run([HttpTrigger(WebHookType = "genericJson")]HttpRequestMessage req, TraceWriter log)
{
log.Info($"Change base 64 Encoding function App was triggered");
Encoding inputEncoding = null;
string jsonContent = await req.Content.ReadAsStringAsync();
dynamic data = JsonConvert.DeserializeObject(jsonContent);
if (data == null || data.text == null || data.encodingInput == null || data.encodingOutput == null)
{
return req.CreateResponse(HttpStatusCode.BadRequest, new
{
error = "Please pass text/encodingOutput properties in the input Json object."
});
}
try
{
string encodingInput = data.encodingInput;
inputEncoding = Encoding.GetEncoding(name: encodingInput);
}
catch (ArgumentException)
{
return req.CreateResponse(HttpStatusCode.BadRequest, new
{
error = "Input char set value '" + data.encodingInput + "' is not supported. Supported value are listed at https://msdn.microsoft.com/en-us/library/system.text.encoding(v=vs.110).aspx."
});
}
Encoding encodingOutput;
try
{
string outputEncoding = data.encodingOutput;
encodingOutput = Encoding.GetEncoding(outputEncoding);
}
catch (ArgumentException)
{
return req.CreateResponse(HttpStatusCode.BadRequest, new
{
error = "Output char set value '" + data.encodingOutput + "' is not supported. Supported value are listed at https://msdn.microsoft.com/en-us/library/system.text.encoding(v=vs.110).aspx."
});
}
string input = data.text;
var outputBytes = Encoding.Convert(srcEncoding: inputEncoding, dstEncoding: encodingOutput, bytes: Convert.FromBase64String(input));
var response = req.CreateResponse(HttpStatusCode.OK);
response.Content = new StringContent(content: JsonConvert.SerializeObject(new
{
text = Convert.ToBase64String(outputBytes)
}).ToString(), encoding: encodingOutput, mediaType: "application/json");
return response;
}
}
}
The function app receives the following input and encodes to the desired format, sends it back.
{
"encodingInput": "utf-8",
"encodingOutput": "windows-1252",
"text": "U0hQMDAwMDExMzZ8VHN1YmFraSBFdXJvcGUgQi5WLnxBdmVudHVyaWpufDMzMTYgTEJ8RG9yZHJlY2h0fE5MfDE4NDMwOHxSdXRoZW5iZXJnIExhbmR0ZWNobmlrfENhcmwtQm9yZ3dhcmQtU3RyIDF8R8O8dGVyc2xvaHwzMzMzNXxERXwyMDIwMDQyOXw0OE58U3xCb3ggM3xDaGFpbiBhbmQgcGFydHN8M3wwLjAyfA=="
}If the above approach doesn’t work, then please use the following approach, which gave me the desired results.
using System;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Azure.WebJobs.Host;
using Newtonsoft.Json;
namespace PnJ.FunctionApp.ConvertEncoding
{
public static class ChangeEncoding
{
[FunctionName("ChangeEncoding")]
public static async Task<HttpResponseMessage> Run([HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)]HttpRequestMessage req, TraceWriter log)
{
log.Info($"Change Encoding function App was triggered was triggered!");
Encoding inputEncoding = null;
string jsonContent = await req.Content.ReadAsStringAsync();
dynamic data = JsonConvert.DeserializeObject(jsonContent);
if (data == null || data.text == null || data.encodingInput == null || data.encodingOutput == null)
{
return req.CreateResponse(HttpStatusCode.BadRequest, new
{
error = "Please pass text/encodingOutput properties in the input Json object."
});
}
try
{
string encodingInput = data.encodingInput;
inputEncoding = Encoding.GetEncoding(name: encodingInput);
}
catch (ArgumentException)
{
return req.CreateResponse(HttpStatusCode.BadRequest, new
{
error = "Input char set value '" + data.encodingInput + "' is not supported. Supported value are listed at https://msdn.microsoft.com/en-us/library/system.text.encoding(v=vs.110).aspx."
});
}
Encoding encodingOutput = null;
try
{
string outputEncoding = data.encodingOutput;
encodingOutput = Encoding.GetEncoding(outputEncoding);
}
catch (ArgumentException)
{
return req.CreateResponse(HttpStatusCode.BadRequest, new
{
error = "Output char set value '" + data.encodingOutput + "' is not supported. Supported value are listed at https://msdn.microsoft.com/en-us/library/system.text.encoding(v=vs.110).aspx."
});
}
string input = data.text;
var outputBytes = Encoding.Convert(srcEncoding: inputEncoding, dstEncoding: encodingOutput, inputEncoding.GetBytes(input));
var response = req.CreateResponse(HttpStatusCode.OK);
MemoryStream ms = new MemoryStream(outputBytes);
response.Content = new StreamContent(ms);
response.Content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
return response;
}
}
}
{
"encodingInput": "utf-8",
"encodingOutput": "windows-1252",
"text": "U0hQMDAwMDExMzZ8VHN1YmFraSBFdXJvcGUgQi5WLnxBdmVudHVyaWpufDMzMTYgTEJ8RG9yZHJlY2h0fE5MfDE4NDMwOHxSdXRoZW5iZXJnIExhbmR0ZWNobmlrfENhcmwtQm9yZ3dhcmQtU3RyIDF8R8O8dGVyc2xvaHwzMzMzNXxERXwyMDIwMDQyOXw0OE58U3xCb3ggM3xDaGFpbiBhbmQgcGFydHN8M3wwLjAyfA=="
}I created following Azure to Logic App to test the out come. The outcome from the second approach gave me better results.
- Azure Integration: Generate a Flat file with ANSI encoding using Logic App
- Azure Integration: Setting null value in Logic App Action to Dynamics 365
- Azure Integration: Tracking, Exception Handling and Monitoring of Azure Logic App
- Azure Integration Release Management best practices
- Azure Integration: ARM visualiser on Azure portal
- Azure: Enumerating all the Logic App runs history using PowerShell and REST API
