Running MS Orleans on AWS ECS

We use MS Orleans framework for one of our projects recently. We host our services on AWS ECS. This creates some problems for us as Orleans silos need to be able to communicate to each other and there would be some networking issues when they try to talk to each other due to multiple docker instances cannot talk to each other in ECS’s network by default. This means we cannot run multiple docker instances and can only run one instance. In this article, I’m going to show how we get this working.

The solution is to utilise Amazon ECS Container Metadata to find the host’s port and IP which we assign to Orleans to use as membership.

Enable Amazon ECS Container Metadata

If not yet enabled, follow this instruction to enable Amazon ECS Container Metadata

Adding port mappings

We need to add MS Orleans default silo port (11111) and default gateway port (30000) to instance’s port mappings. See Port Mappings from AWS Documentation

{
  "portMappings": [
    {
      "containerPort": 11111,
      "hostPort": 0,
      "protocol": "tcp"
    },
    {
      "containerPort": 30000,
      "hostPort": 0,
      "protocol": "tcp"
    }
  ]
}

Getting ports and container instance IP address

We’re going to add a service to retrieve the port information from Amazon Task ECS Metadata endpoint. We’re also going to have a function to retrieve the local IPv4 from the instance metadata endpoint http://169.254.169.254/latest/meta-data/local-ipv4

    public class ContainerMetadataClient
    {
        private readonly ILogger _logger;
        private readonly HttpClient _httpClient;

        public ContainerMetadataClient(ILogger logger)
        {
            _logger = logger;
            var handler = new SocketsHttpHandler
            {
                PooledConnectionLifetime = TimeSpan.FromMinutes(15) // Recreate every 15 minutes
            };

            _httpClient = new HttpClient(handler);
        }

        public async Task<ContainerResponse?> GetContainerMetadata()
        {
            try
            {
                _logger.LogInformation("Fetching ECS Metadata response");

                var containerMetadataUriEnvironmentVariable = Environment.GetEnvironmentVariable("ECS_CONTAINER_METADATA_URI_V4");

                if (Uri.TryCreate(containerMetadataUriEnvironmentVariable, UriKind.Absolute, out var containerMetadataUri))
                {
                    var response = await _httpClient.GetAsync(containerMetadataUri);
                    response.EnsureSuccessStatusCode();

                    var containerMetadataString = await response.Content.ReadAsStringAsync();

                    _logger.LogInformation("ECS Metadata response: {@response}", containerMetadataString);
                    if (!string.IsNullOrEmpty(containerMetadataString))
                    {
                        return JsonConvert.DeserializeObject<ContainerResponse>(containerMetadataString);
                    }
                }
            }
            catch (WebException ex) when (ex.Status == WebExceptionStatus.UnknownError)
            {
                _logger.LogWarning(ex, "Network is unreachable.");
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Failed to get AWS ECS container metadata.");
            }

            return default;
        }

        public async Task<IPAddress?> GetHostPrivateIPv4Address()
        {
            try
            {
                _logger.LogInformation("Fetching ECS private IPv4 response");

                var token = await GetAccessToken();
                if (token == null)
                {
                    return default;
                }

                var request = new HttpRequestMessage(
                    HttpMethod.Get,
                    new Uri("http://169.254.169.254/latest/meta-data/local-ipv4"));
                request.Headers.Add("X-aws-ec2-metadata-token", token);

                var response = await _httpClient.SendAsync(request);
                response.EnsureSuccessStatusCode();

                var result = await response.Content.ReadAsStringAsync();
                _logger.LogInformation("ECS private IPv4 response: {@response}", result);

                return IPAddress.Parse(result);
            }
            catch (WebException ex) when (ex.Status == WebExceptionStatus.UnknownError)
            {
                _logger.LogWarning(ex, "Network is unreachable.");
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Failed to get AWS ECS private IPv4.");
            }

            return default;
        }

        private async Task<string?> GetAccessToken()
        {
            try
            {
                _logger.LogInformation("Fetching ECS access token");

                var request = new HttpRequestMessage(
                    HttpMethod.Put,
                    new Uri("http://169.254.169.254/latest/api/token"));
                request.Headers.Add("X-aws-ec2-metadata-token-ttl-seconds", "21600");

                var response = await _httpClient.SendAsync(request);
                response.EnsureSuccessStatusCode();

                var result = await response.Content.ReadAsStringAsync();

                _logger.LogInformation("Fetching ECS access token successfully");
                return result;
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Failed to get ECS access token.");
            }

            return default;
        }
    }

    public class ContainerResponse
    {
        public List<PortResponse> Ports { get; set; } = new();
    }

    public class PortResponse
    {
        public ushort? ContainerPort { get; set; }
        public string Protocol { get; set; } = string.Empty;
        public ushort? HostPort { get; set; }
    }

Configure MS Orleans endpoint

Configure the endpoint part like this

.Configure<EndpointOptions>(options =>
{
    using ILoggerFactory factory = LoggerFactory.Create(builder => builder.AddConsole());
    ILogger logger = factory.CreateLogger(nameof(ContainerMetadataClient));

    var containerMetadataClient = new ContainerMetadataClient(logger);

    var awsContainerMetadata = containerMetadataClient.GetContainerMetadata().Result;
    var siloPort = awsContainerMetadata?.Ports?.FirstOrDefault(p => p.ContainerPort == EndpointOptions.DEFAULT_SILO_PORT)?.HostPort ?? EndpointOptions.DEFAULT_SILO_PORT;
    var gatewayPort = awsContainerMetadata?.Ports?.FirstOrDefault(p => p.ContainerPort == EndpointOptions.DEFAULT_GATEWAY_PORT)?.HostPort ?? EndpointOptions.DEFAULT_GATEWAY_PORT;
    var advertisedIPAddress = containerMetadataClient.GetHostPrivateIPv4Address().Result ?? Dns.GetHostAddresses(Dns.GetHostName()).First();

    options.AdvertisedIPAddress = advertisedIPAddress;
    options.SiloPort = siloPort;
    options.GatewayPort = gatewayPort;
    options.GatewayListeningEndpoint = new IPEndPoint(IPAddress.Any, EndpointOptions.DEFAULT_GATEWAY_PORT);
    options.SiloListeningEndpoint = new IPEndPoint(IPAddress.Any, EndpointOptions.DEFAULT_SILO_PORT);
})

That’s it. The solution is quite straightforward and I hope it helps you.