Redis

Redis4.0新特性

PSYNC2

在2.8到4.0之间的版本中，Redis同步机制为PSYNC1，在4.0中升级为PSYNC2。其中主要优化减少了全量同步的操作。比如下列场景：

  1. 在之前的版本中，主节点重启导致run id发生变化，从而导致从节点获取同步状态信息错误，必须重新拉取主节点的全量数据，如果此时主节点数据量过大，将可能同步失败并带来消耗大量资源性能 
    1
2
3
4
5
6
7
8
9

    32716:M 01 Apr 16:58:50.209 * The server is now ready to accept connections on port 7379
32716:M 01 Apr 16:58:51.006 * Slave 10.0.139.162:7379 asks for synchronization
32716:M 01 Apr 16:58:51.006 * Partial resynchronization not accepted: Runid mismatch (Client asked for runid '7648f906ab47aca443a7a07853c6eece6fa50d80', my runid is 'ac3b862f1e65d83a23cc474115f09190bdcfdfdd')
32716:M 01 Apr 16:58:51.006 * Starting BGSAVE for SYNC with target: disk
32716:M 01 Apr 16:58:51.007 * Background saving started by pid 32719
32719:C 01 Apr 16:58:51.009 * DB saved on disk
32719:C 01 Apr 16:58:51.009 * RDB: 102 MB of memory used by copy-on-write
32716:M 01 Apr 16:58:51.010 * Background saving terminated with success
32716:M 01 Apr 16:58:51.010 * Synchronization with slave 10.0.139.162:7379 succeeded
  2. PSYNC1因为网络中断或者阻塞导致主从中断，恢复后也需要从主节点重新同步一份全量数据。PSYNC2只需要同步延迟的数据
  3. 在进行链式复制，例如节点A->节点B->节点C架构，当节点A出现问题，C并不能正常同步B节点，当B节点作为主节点时，C节点需要全量同步一次B节点的数据，PSYNC2解决了这一问题。
  4. 在一主多从架构下，如果主节点故障，另一个节点提升为主节点，PYSYNC2只需要增量向新的主节点同步数据

Lazy Free

Redis4.0新增了Lazy Free特性，可以理解为惰性删除，在删除键的时候，把key释放操作由单独的后台子线程去处理，避免在删除bigkey时对Redis主线程的阻塞问题。在4.0之前想要删除一个bigkey只能通过scan去不断扫描-删除-扫描的循环过程，但如果业务上的bigkey因为超时而自动删除的过程，那这个操作将会导致Redis实例长时间阻塞。

Lazy Free分两种：一种是主动删除，也就是通过DEL等命令来执行删除；一种是TTL过期或超过maxmemory限制。

主动删除

DEL命令的Lazy Free实现是由UNLINK命令来完成的，当键的元素超过64个就会把释放操作，交给单独的bio线程来完成。

1

> UNLINK testlist

FLUSHALL/FLUSHDB在4.0之前是一个极度危险的操作，因为他会长时间阻塞Redis。现在通过添加ASYNC异步清理选项，Redis在清理DB或者实例时，操作是由异步完成的。

1

> flushdb async

被动删除

  • lazyfree-lazy-eviction 针对redis到达maxmemory限制并设置了淘汰策略的场景是否采用lazy free，开启lazy free由于异步可能导致key释放不及时
  • lazyfree-lazy-expire 针对设置了TTL的键超时删除时是否采用lazy free，建议开启
  • lazyfree-lazy-server-del 例如rename之类的命令会隐藏包含DEL命令，开启该参数可以使其采用lazy free机制
  • slave-lazy-flush 针对slave全量同步前会通过flushall清理自身所有数据的场景，可减少全量同步的时间

通过查看lazyfree_pending_objects获取lazy free操作在等待实际回收的键的个数，无法获取单个键的元素大小或内存大小。

混合持久化

4.0之前Redis持久化策略分为RDB持久化和AOF持久化，两者各有其优点，详细参见Redis持久化策略。在4.0中新引入了混合持久化，将两者融合在一起，重写AOF时写入RDB数据并将AOF追加到文件中，使得能够更快地重写以及快速加载。

memory命令

4.0新增的MEMORY命令能够用于分析内存情况，例如单个键的内存，整体实例的内存等。

memory usage

memory usage用来查看单个键的内存，与debug object获取的serializedlength差异较大

1
2
3
4

10.0.139.161:6380> memory usage hello
(integer) 56
10.0.139.161:6380> debug object hello
Value at:0x7f0101a1b080 refcount:1 encoding:embstr serializedlength:6 lru:7262128 lru_seconds_idle:1469235

usage针对集合数据类型采用类似LRU SAMPLES抽样的方式，默认抽样5个元素求平均元素个数，得出内存占用情况，这是一个近似值，抽样的SAMPLES个数可以指定，SAMPLES个数越大，内存越精确，但相应的CPU TIME也会更长

1

> MEMORY usage hkey SAMPLES  1000

memory stats

memory stats能够返回当前实例内存的情况

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

10.0.139.161:6380> memory stats
 1) "peak.allocated"
 2) (integer) 2679160
 3) "total.allocated"
 4) (integer) 2638280
 5) "startup.allocated"
 6) (integer) 1439560
 7) "replication.backlog"
 8) (integer) 1048576
 9) "clients.slaves"
10) (integer) 16858
11) "clients.normal"
12) (integer) 49630
13) "aof.buffer"
14) (integer) 0
15) "db.0"
16) 1) "overhead.hashtable.main"
    2) (integer) 72
    3) "overhead.hashtable.expires"
    4) (integer) 0
17) "overhead.total"
18) (integer) 2554696
19) "keys.count"
20) (integer) 1
21) "keys.bytes-per-key"
22) (integer) 1198720
23) "dataset.bytes"
24) (integer) 83584
25) "dataset.percentage"
26) "6.972771167755127"
27) "peak.percentage"
28) "98.474151611328125"
29) "fragmentation"
30) "3.8314499855041504"
  • peak.allocated：实例启动以来，分配的内存峰值，同于info memory的used_memory_peak
  • total.allocated：当前分配的内存字节数，同于info memory的used_memory
  • startup.allocated：Redis启动使用的内存字节数
  • replication.backlog：Redis复制积压缓冲区的内存字节数，由参数repl-backlog-size控制
  • clients.slaves：所有slave clients消耗的内存字节数
  • clients.normal：普通客户端消耗的内存字节数
  • aof.buffer：AOF Buffer使用的内存字节数
  • keys.count：实例上key的数量，同于dbsize
  • keys.bytes-per-key：每个key平均占用字节数，keys.bytes-per-key = (total.allocated-startup.allocated)/keys.count
  • dataset.bytes：数据占用的内存字节数，dataset.bytes = total.allocated - overhead.total
  • dataset.percentage：数据内存占用的百分比，dataset.percentage = dataset.bytes/(total.allocated-startup.allocated) * 100%
  • peak.percentage：当前内存和峰值内存的占比，peak.percentage = total.allocated/peak.allocated * 100%
  • fragmentation：内存碎片率

memory doctor

memory docter分析redis内存使用情况，给出一些诊断意见

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

10.0.139.161:6380> memory doctor
"Sam, I detected a few issues in this Redis instance memory implants:\n\n
Peak memory: In the past this instance used more than 130% the memory that is currently using. 
The allocator is normally not able to release memory after a peak, 
so you can expect to see a big fragmentation ratio,
however this is actually harmless and is only due to the memory peak, and if the Redis instance Resident Set Size (RSS) is currently bigger than expected,
the memory will be used as soon as you fill the Redis instance with more data.
 If the memory peak was only occasional and you want to try to reclaim memory, 
 please try the MEMORY PURGE command, otherwise the only other option is to 
 shutdown and restart the instance.\n\n
 I'm here to keep you safe, Sam. I want to help you.\n"

doctor主要分析下列情况，针对给出意见：

  • used_memory小于5M，不进行分析
  • 峰值内存大于当前allocated的内存1.5倍，说明RSS远大于used_memory
  • 内存碎片率大于1.4
  • 普通客户端平均使用内存大于200KB
  • slave客户端平均使用内存大于10MB

memory purge

memory purge通过调用jemalloc进行内存释放，即释放内存碎片。当Redis删除了大量的key时，redis并不会将这部分内存回收还给操作系统，产生了大量内存碎片，4.0中对于碎片处理提供了memory purge和Active memory defragmentation两种方式。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

10.0.139.161:6380> info memory
# Memory
used_memory:2679232
used_memory_human:2.56M
used_memory_rss:10108928
used_memory_rss_human:9.64M
used_memory_peak:2679232
used_memory_peak_human:2.56M
used_memory_peak_perc:100.00%
used_memory_overhead:2587468
used_memory_startup:1439560
used_memory_dataset:91764
used_memory_dataset_perc:7.40%
total_system_memory:8185929728
total_system_memory_human:7.62G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:24576000000
maxmemory_human:22.89G
maxmemory_policy:noeviction
mem_fragmentation_ratio:3.77
mem_allocator:jemalloc-4.0.3
active_defrag_running:0
lazyfree_pending_objects:0
10.0.139.161:6380> memory purge
OK
10.0.139.161:6380> info memory
# Memory
used_memory:2638496
used_memory_human:2.52M
used_memory_rss:10031104
used_memory_rss_human:9.57M
used_memory_peak:2679232
used_memory_peak_human:2.56M
used_memory_peak_perc:98.48%
used_memory_overhead:2554696
used_memory_startup:1439560
used_memory_dataset:83800
used_memory_dataset_perc:6.99%
total_system_memory:8185929728
total_system_memory_human:7.62G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:24576000000
maxmemory_human:22.89G
maxmemory_policy:noeviction
mem_fragmentation_ratio:3.80
mem_allocator:jemalloc-4.0.3
active_defrag_running:0
lazyfree_pending_objects:0

memory malloc-stats

此命令用于打印allocator内部的状态，目前只支持jemalloc

参考链接

  1. Memory Command
  2. Redis4.0 Lazy Free
  3. Redis4.0 RELEASE NOTES
comments powered by Disqus