Azure webjob to check for files on an FTP server

In my last project, the client wants their files, once uploaded to the FTP server, processed and imported into their database. Previously, I would implement a windows service and install it on the FTP server. However, since the FTP server is hosted on Azure as a virtual machine so I decided to implement an Azure webjob instead as it is easier to deploy as part of the website deployment. I could also have used Azure Function but I haven’t tried it yet so maybe it’s something for next time I build something like this. You can read more about Azure Function here.

Firstly, create a new project in Visual Studio -> Select Azure Webjob template. You might need to install it if you don’t already cloud templates installed.

Then, create a client to connect to FTP server. The client requires the address to the FTP server, username and password to authenticate with the server and an SSL certificate if required. It has a private function CreateFtpWebRequest which creates the request to the FTP server. It requires the file/directory location on the server and the action (WebRequestMethods.Ftp) to be performed to that file/directory. You can find more details on FTP actions here.

    public class FtpClient
    {
        private readonly string address;
        private readonly string password;
        private readonly string sslCert;
        private readonly string userName;

        public FtpClient(string userName, string password, string address, string sslCert = null)
        {
            this.userName = userName;
            this.password = password;
            this.address = address;
            this.sslCert = sslCert;
        }

        public void DeleteFile(string fileName)
        {
            var request = this.CreateFtpWebRequest(fileName, WebRequestMethods.Ftp.DeleteFile);
            request.GetResponse();
        }

        public string[] GetDirectoryListing(string directory)
        {
            var request = this.CreateFtpWebRequest(directory, WebRequestMethods.Ftp.ListDirectory);

            using (var directoryListResponse = (FtpWebResponse)request.GetResponse())
            {
                using (var directoryListResponseReader = new StreamReader(directoryListResponse.GetResponseStream()))
                {
                    var responseString = directoryListResponseReader.ReadToEnd();
                    var results = responseString.Split(new[] { "\r\n", "\n" }, StringSplitOptions.RemoveEmptyEntries);
                    return results;
                }
            }
        }

        public Stream GetFile(string fileName)
        {
            var request = this.CreateFtpWebRequest(fileName, WebRequestMethods.Ftp.DownloadFile);

            using (var responseStream = request.GetResponse().GetResponseStream())
            {
                if (responseStream == null)
                {
                    throw new FileNotFoundException($"File not found: {fileName}.");
                }

                var stream = new MemoryStream();
                responseStream.CopyTo(stream);
                return stream;
            }
        }

        public void MoveFile(string source, string destination, Stream fileStream)
        {
            this.UploadFile(destination, fileStream);
            this.DeleteFile(source);
        }

        public void UploadFile(string destination, Stream fileStream)
        {
            var request = this.CreateFtpWebRequest(destination, WebRequestMethods.Ftp.UploadFile);

            using (var sw = request.GetRequestStream())
            {
                fileStream.Seek(0, SeekOrigin.Begin);
                fileStream.CopyTo(sw);
            }

            request.GetResponse();
        }

        private FtpWebRequest CreateFtpWebRequest(string directory, string method)
        {
            var request = (FtpWebRequest)WebRequest.Create($"{this.address}/{directory}");
            request.Method = method;
            request.Credentials = new NetworkCredential(this.userName, this.password);

            if (!string.IsNullOrWhiteSpace(this.sslCert))
            {
                request.EnableSsl = true;

                var cert = X509Certificate.CreateFromCertFile(this.sslCert);
                var certCollection = new X509CertificateCollection { cert };

                request.ClientCertificates = certCollection;
            }

            return request;
        }

In the Functions.cs, replace the default ProcessQueueMessage with the following. The reason is the webjob is triggered on a schedule rather than automatically when there’s a queue message.

        [NoAutomaticTrigger]
        public void ProcessFiles(TextWriter log)
        {
            // Logics goes here
        }

The TextWriter will be injected automatically by Azure Webjob when it’s run. The full function is below.

        [NoAutomaticTrigger]
        public void ProcessFiles(TextWriter log)
        {
            var ftpClient = new FtpClient("username", "password", "ftpLocation", "SSLCertificateLocation");
            var dataFileDirectory = "WatchFolder/CSVFiles";
            
            var dispensingDirectoryListing = ftpClient.GetDirectoryListing(dataFileDirectory);
            var csvFiles = dispensingDirectoryListing.Where(x => x.EndsWith(".csv", StringComparison.OrdinalIgnoreCase)).ToList();
            log.WriteLine($"Found {csvFiles.Count} new file(s).");

            foreach (var fileName in csvFiles)
            {
                // File stream processing logic goes here
            }            
        }

Finally in Program.cs, you need to tell it to call the ProcessFiles in Function.cs since it is not an automatic triggered function.

        static void Main()
        {
            var host = new JobHost();
            host.Call(typeof(Functions).GetMethod("ProcessFiles"));

            host.RunAndBlock();
        }