Dears,
I tried to access remote redis cache services from redis sentinel services for high availability. But sentinel server couldnt resolve redis host name.
java.net.UnknownHostException: No such host is known (redis-0.redis.xxxx-service-platform.svc.cluster.local) at java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java) at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929) ... 38 frames excluded Wrapped by: io.lettuce.core.RedisConnectionException: Unable to connect to redis-0.redis.xxxx-service-platform.svc.cluster.local:6379 at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:78) at io.lettuce.core.RedisConnectionException.create(RedisConnectionException.java:56) ... 6 frames excluded ... 28 common frames omitted
Also I see above the log on sentinel services like as the follows
1:X 12 Sep 2023 21:56:50.711 # Failed to resolve hostname 'redis-0.redis.xxxx-service-platform.svc.cluster.local' 1:X 12 Sep 2023 21:56:51.773 # Failed to resolve hostname 'redis-0.redis.xxxx-service-platform.svc.cluster.local' 1:X 12 Sep 2023 21:56:52.806 # Failed to resolve hostname 'redis-0.redis.xxxx-service-platform.svc.cluster.local' 1:X 12 Sep 2023 21:56:53.838 # Failed to resolve hostname 'redis-0.redis.xxxx-service-platform.svc.cluster.local' 1:X 12 Sep 2023 21:56:54.908 # Failed to resolve hostname 'redis-0.redis.xxxx-service-platform.svc.cluster.local' 1:X 12 Sep 2023 21:56:55.914 # Failed to resolve hostname 'redis-0.redis.xxxx-service-platform.svc.cluster.local' 1:X 12 Sep 2023 21:56:56.925 # Failed to resolve hostname 'redis-0.redis.xxxx-service-platform.svc.cluster.local' 1:X 12 Sep 2023 21:56:57.967 # Failed to resolve hostname 'redis-0.redis.xxxx-service-platform.svc.cluster.local' 1:X 12 Sep 2023 21:56:59.037 # Failed to resolve hostname 'redis-0.redis.xxxx-service-platform.svc.cluster.local'
Could you help me how to solve the problem?
Thanks,
I tried too much configuration but they didnt work, Sentinel and redis works the same cluster but on remote server, but sentinel should solve redis hostname
apiVersion: v1kind: ConfigMapmetadata: name: redis-config-mapdata: # Variable needed for sentinel init container REDIS_NODES: "redis-0.redis,redis-1.redis,redis-2.redis" # Config used in all redises redis.conf: | # Redis configuration file example. # ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ bind 0.0.0.0 # Protected mode is a layer of security protection, in order to avoid that # Close the connection after a client is idle for N seconds (0 to disable) timeout 300 # A reasonable value for this option is 300 seconds, which is the new # Redis default starting with Redis 3.2.1. tcp-keepalive 30 # Configure allowed ciphers. See the ciphers(1ssl) manpage for more information # about the syntax of this string. # # Note: this configuration applies only to <= TLSv1.2. # # tls-ciphers DEFAULT:!MEDIUM # Configure allowed TLSv1.3 ciphersuites. See the ciphers(1ssl) manpage for more # information about the syntax of this string, and specifically for TLSv1.3 # ciphersuites. # # tls-ciphersuites TLS_CHACHA20_POLY1305_SHA256 # When choosing a cipher, use the server's preference instead of the client # preference. By default, the server follows the client's preference. # verbose (many rarely useful info, but not a mess like the debug level) # notice (moderately verbose, what you want in production probably) # warning (only very important / critical messages are logged) loglevel notice # Specify the log file name. Also the empty string can be used to force # Redis to log on the standard output. Note that if you use standard # output for logging but daemonize, logs will be sent to /dev/null logfile "" # To enable logging to the system logger, just set 'syslog-enabled' to yes, # and optionally update the other syslog parameters to suit your needs. # syslog-enabled no # Specify the syslog identity. # syslog-ident redis # Specify the syslog facility. Must be USER or between LOCAL0-LOCAL7. # syslog-facility local0 # Set the number of databases. The default database is DB 0, you can select # ASCII art logo in startup logs by setting the following option to yes. always-show-logo yes ################################ SNAPSHOTTING ################################ # # RDB files created with checksum disabled have a checksum of zero that will # tell the loading code to skip the check. rdbchecksum yes # The filename where to dump the DB # dbfilename "dump.rdb" # Remove RDB files used by replication in instances without persistence # enabled. By default this option is disabled, however there are environments # where for regulations or other security concerns, RDB files persisted on # disk by masters in order to feed replicas, or stored on disk by replicas # # Note that you must specify a directory here, not a file name. dir "/data" # When a replica loses its connection with the master, or when the replication # With slow disks and fast (large bandwidth) networks, diskless replication # works better. repl-diskless-sync no # When diskless replication is enabled, it is possible to configure the delay repl-diskless-sync-delay 5 repl-diskless-load disabled # Replicas send PINGs to server in a predefined interval. It's possible to # change this interval with the repl_ping_replica_period option. The default # value is 10 seconds. # appendonly no # repl-ping-replica-period 10 # The following option sets the replication timeout for: # # It is important to make sure that this value is greater than the value # specified for repl-ping-replica-period otherwise a timeout will be detected # every time there is low traffic between the master and the replica. # # repl-timeout 60 # Linux kernels using a default configuration. # # If you select "no" the delay for data to appear on the replica side will # be reduced but more bandwidth will be used for replication. # # By default we optimize for low latency, but in very high traffic conditions # or when the master and replicas are many hops away, turning this to "yes" may # be a good idea. repl-disable-tcp-nodelay no # Set the replication backlog size. The backlog is a buffer that accumulates # replica data when replicas are disconnected for some time, so that when a # replica wants to reconnect again, often a full resync is not needed, but a # partial resync is enough, just passing the portion of data the replica # missed while disconnected. # # The bigger the replication backlog, the longer the time the replica can be # disconnected and later be able to perform a partial resynchronization. # # The backlog is only allocated once there is at least a replica connected. # # repl-backlog-size 1mb # After a master has no longer connected replicas for some time, the backlog # will be freed. The following option configures the amount of seconds that # need to elapse, starting from the time the last replica disconnected, for # the backlog buffer to be freed. # # Note that replicas never free the backlog for timeout, since they may be # promoted to masters later, and should be able to correctly "partially # resynchronize" with the replicas: hence they should always accumulate backlog. # # A value of 0 means to never release the backlog. # # repl-backlog-ttl 3600 # The replica priority is an integer number published by Redis in the INFO # output. It is used by Redis Sentinel in order to select a replica to promote # into a master if the master is no longer working correctly. # # A replica with a low priority number is considered better for promotion, so # for instance if there are three replicas with priority 10, 100, 25 Sentinel # will pick the one with priority 10, that is the lowest. # # However a special priority of 0 marks the replica as not able to perform the # role of master, so a replica with priority of 0 will never be selected by # Redis Sentinel for promotion. # # By default the priority is 100. replica-priority 100 # It is possible for a master to stop accepting writes if there are less than # N replicas connected, having a lag less or equal than M seconds. # # The N replicas need to be in "online" state. # # The lag in seconds, that must be <= the specified value, is calculated from # the last ping received from the replica, that is usually sent every second. # # This option does not GUARANTEE that N replicas will accept the write, but # will limit the window of exposure for lost writes in case not enough replicas # are available, to the specified number of seconds. # # For example to require at least 3 replicas with a lag <= 10 seconds use: # # min-replicas-to-write 3 # min-replicas-max-lag 10 # # Setting one or the other to 0 disables the feature. # # By default min-replicas-to-write is set to 0 (feature disabled) and # min-replicas-max-lag is set to 10. # A Redis master is able to list the address and port of the attached # replicas in different ways. For example the "INFO replication" section # offers this information, which is used, among other tools, by # Redis Sentinel in order to discover replica instances. # Another place where this info is available is in the output of the # "ROLE" command of a master. # # The listed IP and address normally reported by a replica is obtained # in the following way: # # IP: The address is auto detected by checking the peer address # of the socket used by the replica to connect with the master. # # Port: The port is communicated by the replica during the replication # handshake, and is normally the port that the replica is using to # listen for connections. # # However when port forwarding or Network Address Translation (NAT) is # used, the replica may be actually reachable via different IP and port # pairs. The following two options can be used by a replica in order to # report to its master a specific set of IP and port, so that both INFO # and ROLE will report those values. # # There is no need to use both the options if you need to override just # the port or the IP address. # # replica-announce-ip 5.5.5.5 # replica-announce-port 1234 ############################### KEYS TRACKING ################################# # Redis implements server assisted support for client side caching of values. # This is implemented using an invalidation table that remembers, using # 16 millions of slots, what clients may have certain subsets of keys. In turn # this is used in order to send invalidation messages to clients. Please # to understand more about the feature check this page: # # https://redis.io/topics/client-side-caching # # When tracking is enabled for a client, all the read only queries are assumed # to be cached: this will force Redis to store information in the invalidation # table. When keys are modified, such information is flushed away, and # invalidation messages are sent to the clients. However if the workload is # heavily dominated by reads, Redis could use more and more memory in order # to track the keys fetched by many clients. # # For this reason it is possible to configure a maximum fill value for the # invalidation table. By default it is set to 1M of keys, and once this limit # is reached, Redis will start to evict keys in the invalidation table # even if they were not modified, just to reclaim memory: this will in turn # force the clients to invalidate the cached values. Basically the table # maximum size is a trade off between the memory you want to spend server # side to track information about who cached what, and the ability of clients # to retain cached objects in memory. # # If you set the value to 0, it means there are no limits, and Redis will # retain as many keys as needed in the invalidation table. # In the "stats" INFO section, you can find information about the number of # keys in the invalidation table at every given moment. # # Note: when key tracking is used in broadcasting mode, no memory is used # in the server side so this setting is useless. # # tracking-table-max-keys 1000000 ################################## SECURITY ################################### # Warning: since Redis is pretty fast an outside user can try up to # 1 million passwords per second against a modern box. This means that you # should use very strong passwords, otherwise they will be very easy to break. # Note that because the password is really a shared secret between the client # and the server, and should not be memorized by any human, the password # can be easily a long string from /dev/urandom or whatever, so by using a # long and unguessable password no brute force attack will be possible. # Redis ACL users are defined in the following format: # # user <username> ... acl rules ... # # For example: # # user worker +@list +@connection ~jobs:* on >ffa9203c493aa99 # # The special username "default" is used for new connections. If this user # has the "nopass" rule, then new connections will be immediately authenticated # as the "default" user without the need of any password provided via the # AUTH command. Otherwise if the "default" user is not flagged with "nopass" # the connections will start in not authenticated state, and will require # AUTH (or the HELLO command AUTH option) in order to be authenticated and # start to work. # # The ACL rules that describe what an user can do are the following: # # on Enable the user: it is possible to authenticate as this user. # off Disable the user: it's no longer possible to authenticate # with this user, however the already authenticated connections # will still work. # +<command> Allow the execution of that command # -<command> Disallow the execution of that command # +@<category> Allow the execution of all the commands in such category # with valid categories are like @admin, @set, @sortedset, ... # and so forth, see the full list in the server.c file where # the Redis command table is described and defined. # The special category @all means all the commands, but currently # present in the server, and that will be loaded in the future # via modules. # +<command>|subcommand Allow a specific subcommand of an otherwise # disabled command. Note that this form is not # allowed as negative like -DEBUG|SEGFAULT, but # only additive starting with "+". # allcommands Alias for +@all. Note that it implies the ability to execute # all the future commands loaded via the modules system. # nocommands Alias for -@all. # ~<pattern> Add a pattern of keys that can be mentioned as part of # commands. For instance ~* allows all the keys. The pattern # is a glob-style pattern like the one of KEYS. # It is possible to specify multiple patterns. # allkeys Alias for ~* # resetkeys Flush the list of allowed keys patterns. # ><password> Add this passowrd to the list of valid password for the user. # For example >mypass will add "mypass" to the list. # This directive clears the "nopass" flag (see later). # <<password> Remove this password from the list of valid passwords. # nopass All the set passwords of the user are removed, and the user # is flagged as requiring no password: it means that every # password will work against this user. If this directive is # used for the default user, every new connection will be # immediately authenticated with the default user without # any explicit AUTH command required. Note that the "resetpass" # directive will clear this condition. # resetpass Flush the list of allowed passwords. Moreover removes the # "nopass" status. After "resetpass" the user has no associated # passwords and there is no way to authenticate without adding # some password (or setting it as "nopass" later). # reset Performs the following actions: resetpass, resetkeys, off, # -@all. The user returns to the same state it has immediately # after its creation. # # ACL rules can be specified in any order: for instance you can start with # passwords, then flags, or key patterns. However note that the additive # and subtractive rules will CHANGE MEANING depending on the ordering. # For instance see the following example: # # user alice on +@all -DEBUG ~* >somepassword # # This will allow "alice" to use all the commands with the exception of the # DEBUG command, since +@all added all the commands to the set of the commands # alice can use, and later DEBUG was removed. However if we invert the order # of two ACL rules the result will be different: # # user alice on -DEBUG +@all ~* >somepassword # # Now DEBUG was removed when alice had yet no commands in the set of allowed # commands, later all the commands are added, so the user will be able to # execute everything. # # Basically ACL rules are processed left-to-right. # # For more information about ACL configuration please refer to # the Redis web site at https://redis.io/topics/acl # ACL LOG # # The ACL Log tracks failed commands and authentication events associated # with ACLs. The ACL Log is useful to troubleshoot failed commands blocked # by ACLs. The ACL Log is stored in memory. You can reclaim memory with # ACL LOG RESET. Define the maximum entry length of the ACL Log below. acllog-max-len 128 # Using an external ACL file # # Instead of configuring users here in this file, it is possible to use # a stand-alone file just listing users. The two methods cannot be mixed: # if you configure users here and at the same time you activate the exteranl # ACL file, the server will refuse to start. # # The format of the external ACL user file is exactly the same as the # format that is used inside redis.conf to describe users. # # aclfile /etc/redis/users.acl # IMPORTANT NOTE: starting with Redis 6 "requirepass" is just a compatiblity # layer on top of the new ACL system. The option effect will be just setting # the password for the default user. Clients will still authenticate using # AUTH <password> as usually, or more explicitly with AUTH default <password> # if they follow the new protocol: both will work. #---apiVersion: v1kind: ConfigMapmetadata: name: redis-scripts-config-mapdata: sentinel_init.sh: | #!/bin/bash REDIS_NODES= "redis-0.redis,redis-1.redis,redis-2.redis" for i in ${REDIS_NODES//,/ } do echo "finding master at $i" MASTER=$(redis-cli --no-auth-warning --raw -h $i -a ${REDIS_PASSWORD} info replication | awk '{print $1}' | grep master_host: | cut -d ":" -f2) if [ "${MASTER}" == "" ]; then echo "no master found" MASTER= else echo "found ${MASTER}" break fi done POD_FQDN=$(hostname -f) echo "sentinel monitor mymaster ${MASTER} 6379 2" >> /tmp/master echo "port 5000 $(cat /tmp/master) protected-mode no sentinel resolve-hostnames yes sentinel announce-hostnames yes sentinel announce-ip $POD_FQDN sentinel announce-port 5000 sentinel down-after-milliseconds mymaster 1000 sentinel failover-timeout mymaster 60000 sentinel parallel-syncs mymaster 1 sentinel sentinel-pass ${REDIS_PASSWORD} sentinel auth-pass mymaster ${REDIS_PASSWORD} requirepass ${REDIS_PASSWORD}" > /etc/redis/sentinel.conf cat /etc/redis/sentinel.conf redis_init.sh: | #!/bin/bash cp /tmp/redis/redis.conf /etc/redis/redis.conf echo "requirepass ${REDIS_PASSWORD}" >> /etc/redis/redis.conf echo "masterauth ${REDIS_PASSWORD}" >> /etc/redis/redis.conf echo "replica-announce-ip ${HOSTNAME}.redis" >> /etc/redis/redis.conf echo "replica-announce-port 6379 " >> /etc/redis/redis.conf echo "finding master..." if [ "$(timeout 5 redis-cli -h sentinel -p 5000 -a ${REDIS_PASSWORD} ping)" != "PONG" ]; then echo "sentinel not found, defaulting to redis-0" if [ ${HOSTNAME} == "redis-0" ]; then echo "this is redis-0, not updating config..." else echo "updating redis.conf..." echo "repl-ping-replica-period 3" >> /etc/redis/redis.conf echo "slave-read-only no" >> /etc/redis/redis.conf echo "slaveof redis-0.redis 6379" >> /etc/redis/redis.conf fi else echo "sentinel found, finding master" MASTER="$(redis-cli -h sentinel -p 5000 -a ${REDIS_PASSWORD} sentinel get-master-addr-by-name mymaster | grep -E '(^redis-*)|([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})')" if [ "${HOSTNAME}.redis" == ${MASTER} ]; then echo "this is master, not updating config..." else echo "master found : ${MASTER}, updating redis.conf" echo "slave-read-only no" >> /etc/redis/redis.conf echo "slaveof ${MASTER} 6379" >> /etc/redis/redis.conf echo "repl-ping-replica-period 3" >> /etc/redis/redis.conf fi fi---apiVersion: v1kind: Secretmetadata: name: redis-secrettype: Opaquedata: REDIS_PASSWORD: c3BlZWR5R29uemFsZXMyMDIz---# Headless service so sentinel could access redisses using syntax <pod-name>.<service-name>apiVersion: v1kind: Servicemetadata: name: redis labels: app: redis app.kubernetes.io/component: redis app.kubernetes.io/instance: redisspec: clusterIP: None ports: - port: 6379 targetPort: 6379 name: redis selector: app: redis---# Sentinel service used for project pod connectionapiVersion: v1kind: Servicemetadata: name: sentinel-redis labels: app: sentinel app.kubernetes.io/component: sentinel app.kubernetes.io/instance: sentinelspec: type: ClusterIP sessionAffinity: None ports: - port: 5000 targetPort: 5000 name: sentinel selector: app: sentinel---apiVersion: apps/v1kind: StatefulSetmetadata: name: redisspec: serviceName: redis replicas: 3 selector: matchLabels: app: redis template: metadata: labels: app: redis spec: initContainers: - name: config image: 'scorenexus.isbank:8443/bitnami/redis:7.0' env: - name: REDIS_PASSWORD valueFrom: secretKeyRef: name: redis-secret key: REDIS_PASSWORD command: [ "sh", "-c", "/scripts/redis_init.sh" ] volumeMounts: - name: redis-config mountPath: /etc/redis/ - name: config mountPath: /tmp/redis/ - name: init-script mountPath: /scripts/ containers: - name: redis image: 'scorenexus.isbank:8443/bitnami/redis:7.0' command: ["redis-server"] args: ["/etc/redis/redis.conf"] ports: - containerPort: 6379 name: redis volumeMounts: - name: data mountPath: /data - name: redis-config mountPath: /etc/redis/ volumes: - name: data emptyDir: {} - name: redis-config emptyDir: {} - name: init-script configMap: name: redis-scripts-config-map defaultMode: 0777 items: - key: redis_init.sh path: redis_init.sh - name: config configMap: name: redis-config-map items: - key: redis.conf path: redis.conf---apiVersion: apps/v1kind: StatefulSetmetadata: name: sentinel-redisspec: serviceName: sentinel-redis replicas: 3 selector: matchLabels: app: sentinel template: metadata: labels: app: sentinel spec: initContainers: - name: config image: 'scorenexus.isbank:8443/bitnami/redis:7.0' env: - name: REDIS_NODES valueFrom: configMapKeyRef: name: redis-config-map key: REDIS_NODES - name: REDIS_PASSWORD valueFrom: secretKeyRef: name: redis-secret key: REDIS_PASSWORD command: [ "sh", "-c", "/scripts/sentinel_init.sh" ] volumeMounts: - name: redis-config mountPath: /etc/redis/ - name: init-script mountPath: /scripts/ containers: - name: sentinel image: 'scorenexus.isbank:8443/bitnami/redis:7.0' command: ["redis-sentinel"] args: ["/etc/redis/sentinel.conf"] ports: - containerPort: 5000 name: sentinel volumeMounts: - name: redis-config mountPath: /etc/redis/ - name: data mountPath: /data volumes: - name: init-script configMap: name: redis-scripts-config-map defaultMode: 0777 items: - key: sentinel_init.sh path: sentinel_init.sh - name: redis-config emptyDir: {} - name: data emptyDir: {}---