How to Append a File using Web HDFS REST API in C#?

c# rest api azure hadoop

451 观看

1回复

280 作者的声誉

I am using Azure Data Lake to upload the file before but still want to append the text file content existing Data Lake text file. Is there any option available to append the text file data using Web HDFS REST API in C#?.

I am refer this link enter link description here

Code: I Can refer the above link get the append URL. But how can i use this URL and Append a File using c#?

private const string AppendUrl = "https://{0}.azuredatalakestore.net/webhdfs/v1/{1}?&op=APPEND&noredirect=true";
作者: Saravanakumar Natarajan 的来源 发布者: 2017 年 12 月 27 日

回应 1


2

19257 作者的声誉

决定

If you want to use the Rest Api to do that we could use the following code. I test it with Postman.

private const string AppendUrl = "https://{datalakeName}.azuredatalakestore.net/webhdfs/v1/{filepath}?append=true&op=APPEND&api-version=2016-11-01"
var token = "eyJ0eX.....";
using (var client = new HttpClient())
{
     client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
     var result = client.GetAsync(url).Result;
     var data = result.Content.ReadAsStringAsync().Result;
}

enter image description here

We also could use the Azure Microsoft.Azure.Management.DataLake.Store to do that. How to get the application id and secretkey you could refer to official document. More detail steps to get the permission to access the datalake you could refer to another SO thread.

var applicationId = "application Id";
var secretKey = "secretKey";
var tenantId = "tenant id";
var adlsAccountName = "datalake account name";
var creds = ApplicationTokenProvider.LoginSilentAsync(tenantId, applicationId, secretKey).Result;
var adlsFileSystemClient = new DataLakeStoreFileSystemManagementClient(creds,clientTimeoutInMinutes:60);
var stream = File.OpenRead(@"C:\tom\testtext.txt");
var test = adlsFileSystemClient.FileSystem.AppendWithHttpMessagesAsync(adlsAccountName, "test/abc.txt", stream).Result;

enter image description here

packages:

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Microsoft.Azure.Management.DataLake.Store" version="2.3.0-preview" targetFramework="net452" />
  <package id="Microsoft.Azure.Management.DataLake.StoreUploader" version="1.0.0-preview" targetFramework="net452" />
  <package id="Microsoft.IdentityModel.Clients.ActiveDirectory" version="3.13.8" targetFramework="net452" />
  <package id="Microsoft.Rest.ClientRuntime" version="2.3.9" targetFramework="net452" />
  <package id="Microsoft.Rest.ClientRuntime.Azure" version="3.3.9" targetFramework="net452" />
  <package id="Microsoft.Rest.ClientRuntime.Azure.Authentication" version="2.2.0-preview" targetFramework="net452" />
  <package id="Newtonsoft.Json" version="9.0.2-beta1" targetFramework="net452" />
</packages>
作者: Tom Sun 发布者: 2017 年 12 月 28 日
32x32