上一篇讲了ASM fast mirror resync。此外在12.1这个版本的ASM中,引入了一个新的特性叫做ASM Disk Scrubbing。借用黄老的一句话“scrub对的是corruption,resync对的是stale,两者都是保证一个所谓的data integrity”。本篇主要介绍ASM Disk Scrubbing。
https://docs.oracle.com/database/121/OSTMG/GUID-ECCAE4F3-CFC5-4B55-81D2-FFB6953C035C.htm#OSTMG95351
来自官方文档的介绍
本次要测试3个场景:
- dd破坏数据块,手工触发scrub看是否可以修复
- dd破坏数据块,select看是否会有概率报错,以及ASM是否会尝试修复
- bbed修改其中一个mirror的某行数据,dbv确保不报错,ASM是否修此讹误
创建测试用的磁盘组,为了保证简单,只用2个盘做一个Normal的磁盘组。
1 2 3 4 |
CREATE DISKGROUP SCRUB NORMAL REDUNDANCY DISK '/dev/mapper/data04' SIZE 10240M DISK '/dev/mapper/data05' SIZE 10240M ATTRIBUTE 'compatible.asm'='19.0.0.0','compatible.rdbms'='19.0.0.0','au_size'='4M'; |
之后创建一个1M的表空间,并且在这个表空间里创建一个表并插入数据。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
SQL> alter session set container=ora19pdb1; Session altered. SQL> create tablespace scrub datafile '+SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf' size 1M autoextend off; Tablespace created. SQL> create table tb_scrub (id number,name varchar2(20)) tablespace scrub; Table created. SQL> insert into tb_scrub values(1,'MINOR'); 1 row created. SQL> insert into tb_scrub values(2,'MINOR'); 1 row created. SQL> commit; Commit complete. SQL> select * from tb_scrub; ID NAME ---------- ------------------------------------------------------------ 1 MINOR 2 MINOR SQL> |
确认文件编号:
1 2 3 4 5 6 7 8 9 10 11 12 13 |
SQL> col name for a40 SQL> col type for a20 SQL> SELECT f.group_number, f.file_number, a.name, f.type 2 FROM v$asm_file f, v$asm_alias a 3 WHERE f.group_number=a.group_number and 4 f.file_number=a.file_number 5 and a.name like '%scrub%' 6 ORDER BY 1, 2; GROUP_NUMBER FILE_NUMBER NAME TYPE ------------ ----------- ---------------------------------------- -------------------- 3 256 scrub_001.dbf DATAFILE |
确认文件在ASM中的AU分布:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
SQL> SELECT x.xnum_kffxp "Extent", 2 x.au_kffxp "AU", 3 x.disk_kffxp "Disk #", 4 d.name "Disk name" 5 FROM x$kffxp x, v$asm_disk_stat d 6 WHERE x.group_kffxp=d.group_number 7 and x.disk_kffxp=d.disk_number 8 and x.group_kffxp=3 9 and x.number_kffxp =256 10 ORDER BY 1, 2; Extent AU Disk # Disk name ---------- ---------- ---------- ------------------------------------------------------------------------------------------ 0 26 1 SCRUB_0001 26 0 SCRUB_0000 |
这个数据文件只分配了一个AU,大小是4M,存在两个副本,分别位于两个磁盘上。
把这个文件弄出来。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
[grid@dm01db08 scrub]$ dd if=/dev/mapper/data04 of=0D26AU.dbf bs=4194304 count=1 skip=26 1+0 records in 1+0 records out 4194304 bytes (4.2 MB) copied, 0.0477993 s, 87.7 MB/s [grid@dm01db08 scrub]$ [grid@dm01db08 scrub]$ [grid@dm01db08 scrub]$ [grid@dm01db08 scrub]$ dbv file=0D26AU.dbf DBVERIFY: Release 19.0.0.0.0 - Production on Wed Jun 26 11:16:11 2019 Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved. DBVERIFY - Verification starting : FILE = /home/grid/scrub/0D26AU.dbf DBVERIFY - Verification complete Total Pages Examined : 128 Total Pages Processed (Data) : 5 Total Pages Failing (Data) : 0 Total Pages Processed (Index): 0 Total Pages Failing (Index): 0 Total Pages Processed (Other): 10 Total Pages Processed (Seg) : 0 Total Pages Failing (Seg) : 0 Total Pages Empty : 113 Total Pages Marked Corrupt : 0 Total Pages Influx : 0 Total Pages Encrypted : 0 Highest block SCN : 7291447 (0.7291447) |
确认这个就是数据文件。然后我们dd一个4M的空文件写回去。
首先这里插入一个概念,就是Normal冗余下,每个AU都有一个Primary copy和一个Secondary copy,ASM无论在何种情况下都会去读Primary的copy,当primary不可读时,就去读Secondary的copy,并且将这个copy写回Primary。
A) Block Corruption in Primary Extent
Block reads from disk is affected. Corruption will be found during read from primary extent, we report an ORA-1578 in the ASM alert.log and do a re-read of the same block (primary extent). If the second read fails as well, we read the block from the secondary extent (on a different physical disk) next. This failover to reading from secondary extent happens automatically and transparent for the user/session. If the secondary extent is not corrupt the user will get the results without any error indication other than ORA-1578 about the failed primary extent read in alert.log. Finally ASM will overwrite the corrupt block in the primary extent with the good copy from the secondary. This happens even when the block is unaffected by any transaction. ASM FINDS corrupt blocks in primary extent. That triggers automatically repair at next read.
B) Block Corruption in Secondary Extent
Because block reads always goes to the primary extent first, a corruption in the secondary extent does not have any effect on a read operation. At the next write the block becomes dirty and the new block image will overwrite the old corrupt block on disk. In case of block corruption in the secondary extent there is no diagnostics information available. A corruption in the secondary extent will normally only be seen if the block in the primary extent is also corrupt. ASM fixes corrupt blocks in secondary extent automatically at next write of the block.
所以,我们要破坏Primary的copy,不然看不到坏块触发。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
SQL> select PXN_KFFXP, -- physical extent number \ 2 XNUM_KFFXP, -- virtual extent number 3 DISK_KFFXP, -- disk number 4 AU_KFFXP, -- allocation unit number 5 decode(LXN_KFFXP,0,'Primary',1,'Secondary','header metadata') "AU type" 6 from X$KFFXP 7 where NUMBER_KFFXP=256 -- ASM file 272 8 AND GROUP_KFFXP=3 -- group number 1 9 order by 1; PXN_KFFXP XNUM_KFFXP DISK_KFFXP AU_KFFXP AU type ---------- ---------- ---------- ---------- --------------------------------------------- 0 0 1 26 Primary 1 0 0 26 Secondary |
所以,1号磁盘的26AU是primary,要拿NULL文件去覆盖它:
1 2 3 |
dd of=/dev/mapper/data05 if=/home/grid/scrub/NULL.dbf count=1 conv=notrunc bs=4194304 seek=26 |
随后,我们读表:
1 2 3 4 5 6 7 |
SQL> select * from tb_scrub; ID NAME ---------- ------------------------------------------------------------ 1 MINOR 2 MINOR |
第一次读,会卡顿一下,并且在后台刷出大量日志:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 |
ORA19PDB1(3):Hex dump of (file 50, block 10) in trace file /u01/app/ora19c/diag/rdbms/ora19c/ORA19c/trace/ORA19c_ora_20437.trc ORA19PDB1(3): ORA19PDB1(3):Corrupt block relative dba: 0x0c80000a (file 50, block 10) ORA19PDB1(3):Completely zero block found during buffer read ORA19PDB1(3): ORA19PDB1(3):Reading datafile '+SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf' for corrupt data at rdba: 0x0c80000a (file 50, block 10) ORA19PDB1(3):Read datafile mirror 'SCRUB_0001' (file 50, block 10) found same corrupt data (no logical check) ORA19PDB1(3):Read datafile mirror '' (file 50, block 10) found valid data ORA19PDB1(3):Hex dump of (file 50, block 10) in trace file /u01/app/ora19c/diag/rdbms/ora19c/ORA19c/trace/ORA19c_ora_20437.trc ORA19PDB1(3): ORA19PDB1(3):Repaired corruption at (file 50, block 10) ORA19PDB1(3):Hex dump of (file 50, block 11) in trace file /u01/app/ora19c/diag/rdbms/ora19c/ORA19c/trace/ORA19c_ora_20437.trc ORA19PDB1(3): ORA19PDB1(3):Corrupt block relative dba: 0x0c80000b (file 50, block 11) ORA19PDB1(3):Completely zero block found during multiblock buffer read ORA19PDB1(3): ORA19PDB1(3):Reading datafile '+SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf' for corrupt data at rdba: 0x0c80000b (file 50, block 11) ORA19PDB1(3):Read datafile mirror 'SCRUB_0001' (file 50, block 11) found same corrupt data (no logical check) ORA19PDB1(3):Read datafile mirror '' (file 50, block 11) found valid data ORA19PDB1(3):Hex dump of (file 50, block 11) in trace file /u01/app/ora19c/diag/rdbms/ora19c/ORA19c/trace/ORA19c_ora_20437.trc ORA19PDB1(3): ORA19PDB1(3):Repaired corruption at (file 50, block 11) ORA19PDB1(3):Hex dump of (file 50, block 12) in trace file /u01/app/ora19c/diag/rdbms/ora19c/ORA19c/trace/ORA19c_ora_20437.trc ORA19PDB1(3): ORA19PDB1(3):Corrupt block relative dba: 0x0c80000c (file 50, block 12) ORA19PDB1(3):Completely zero block found during multiblock buffer read ORA19PDB1(3): ORA19PDB1(3):Reading datafile '+SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf' for corrupt data at rdba: 0x0c80000c (file 50, block 12) ORA19PDB1(3):Read datafile mirror 'SCRUB_0001' (file 50, block 12) found same corrupt data (no logical check) ORA19PDB1(3):Read datafile mirror '' (file 50, block 12) found valid data ORA19PDB1(3):Hex dump of (file 50, block 12) in trace file /u01/app/ora19c/diag/rdbms/ora19c/ORA19c/trace/ORA19c_ora_20437.trc ORA19PDB1(3): ORA19PDB1(3):Repaired corruption at (file 50, block 12) ORA19PDB1(3):Hex dump of (file 50, block 13) in trace file /u01/app/ora19c/diag/rdbms/ora19c/ORA19c/trace/ORA19c_ora_20437.trc ORA19PDB1(3): ORA19PDB1(3):Corrupt block relative dba: 0x0c80000d (file 50, block 13) ORA19PDB1(3):Completely zero block found during multiblock buffer read ORA19PDB1(3): ORA19PDB1(3):Reading datafile '+SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf' for corrupt data at rdba: 0x0c80000d (file 50, block 13) ORA19PDB1(3):Read datafile mirror 'SCRUB_0001' (file 50, block 13) found same corrupt data (no logical check) ORA19PDB1(3):Read datafile mirror '' (file 50, block 13) found valid data ORA19PDB1(3):Hex dump of (file 50, block 13) in trace file /u01/app/ora19c/diag/rdbms/ora19c/ORA19c/trace/ORA19c_ora_20437.trc ORA19PDB1(3): ORA19PDB1(3):Repaired corruption at (file 50, block 13) ORA19PDB1(3):Hex dump of (file 50, block 14) in trace file /u01/app/ora19c/diag/rdbms/ora19c/ORA19c/trace/ORA19c_ora_20437.trc ORA19PDB1(3): ORA19PDB1(3):Corrupt block relative dba: 0x0c80000e (file 50, block 14) ORA19PDB1(3):Completely zero block found during multiblock buffer read ORA19PDB1(3): ORA19PDB1(3):Reading datafile '+SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf' for corrupt data at rdba: 0x0c80000e (file 50, block 14) ORA19PDB1(3):Read datafile mirror 'SCRUB_0001' (file 50, block 14) found same corrupt data (no logical check) ORA19PDB1(3):Read datafile mirror '' (file 50, block 14) found valid data ORA19PDB1(3):Hex dump of (file 50, block 14) in trace file /u01/app/ora19c/diag/rdbms/ora19c/ORA19c/trace/ORA19c_ora_20437.trc ORA19PDB1(3): ORA19PDB1(3):Repaired corruption at (file 50, block 14) ORA19PDB1(3):Hex dump of (file 50, block 15) in trace file /u01/app/ora19c/diag/rdbms/ora19c/ORA19c/trace/ORA19c_ora_20437.trc ORA19PDB1(3): ORA19PDB1(3):Corrupt block relative dba: 0x0c80000f (file 50, block 15) ORA19PDB1(3):Completely zero block found during multiblock buffer read ORA19PDB1(3): ORA19PDB1(3):Reading datafile '+SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf' for corrupt data at rdba: 0x0c80000f (file 50, block 15) ORA19PDB1(3):Read datafile mirror 'SCRUB_0001' (file 50, block 15) found same corrupt data (no logical check) ORA19PDB1(3):Read datafile mirror '' (file 50, block 15) found valid data ORA19PDB1(3):Hex dump of (file 50, block 15) in trace file /u01/app/ora19c/diag/rdbms/ora19c/ORA19c/trace/ORA19c_ora_20437.trc ORA19PDB1(3): ORA19PDB1(3):Repaired corruption at (file 50, block 15) |
这里触发的就是ASM Normal冗余情况下,ASM利用另外一个mirror来修复坏块的情况,可以看到它读到错的块,就去读好块,然后repaired。但是注意,这里只修复了10到15号,一共6个坏块。事实上,这个表有8个块,看起来ASM只修复了data block,段头没有修复。
随后,我们利用ASM Disk Scrubbing来修复一下:
1 2 3 4 |
alter diskgroup SCRUB scrub file '+SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf' wait; --此步是分析,不修复 |
注意下面的日志很重要,它表示scrub特性发现了很多坏块,但是,但是,但是,10到15号块是没有,这也证明,ASM的mirror修好了这6个坏块。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 |
2019-06-26T11:42:54.661794+08:00 SQL> alter diskgroup SCRUB scrub file '+SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf' wait 2019-06-26T11:42:54.665810+08:00 NOTE: Waiting for scrubbing to finish 2019-06-26T11:42:54.943761+08:00 NOTE: Corrupted block 1 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf NOTE: Corrupted block 3 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:54.983988+08:00 NOTE: Corrupted block 2 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf NOTE: Corrupted block 5 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.009103+08:00 NOTE: Corrupted block 6 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf NOTE: Corrupted block 7 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.031440+08:00 NOTE: Corrupted block 8 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.056268+08:00 NOTE: Corrupted block 16 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.056586+08:00 NOTE: Corrupted block 9 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.082784+08:00 NOTE: Corrupted block 4 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.165656+08:00 NOTE: Corrupted block 19 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.168293+08:00 NOTE: Corrupted block 18 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.176061+08:00 NOTE: Corrupted block 20 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.194815+08:00 NOTE: Corrupted block 21 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.200156+08:00 NOTE: Corrupted block 22 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.215497+08:00 NOTE: Corrupted block 23 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.222404+08:00 NOTE: Corrupted block 17 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.226144+08:00 NOTE: Corrupted block 24 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.238300+08:00 NOTE: Corrupted block 25 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.245746+08:00 NOTE: Corrupted block 26 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.256011+08:00 NOTE: Corrupted block 27 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.267814+08:00 NOTE: Corrupted block 28 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.280714+08:00 NOTE: Corrupted block 29 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.289762+08:00 NOTE: Corrupted block 30 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.303329+08:00 NOTE: Corrupted block 31 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.313017+08:00 NOTE: Corrupted block 32 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.324851+08:00 NOTE: Corrupted block 33 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.336541+08:00 NOTE: Corrupted block 34 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.349406+08:00 NOTE: Corrupted block 35 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.359935+08:00 NOTE: Corrupted block 36 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.370846+08:00 NOTE: Corrupted block 37 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.383704+08:00 NOTE: Corrupted block 38 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.395228+08:00 NOTE: Corrupted block 39 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.407653+08:00 NOTE: Corrupted block 40 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.419037+08:00 NOTE: Corrupted block 41 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.432278+08:00 NOTE: Corrupted block 42 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.442171+08:00 NOTE: Corrupted block 43 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.453801+08:00 NOTE: Corrupted block 44 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.465083+08:00 NOTE: Corrupted block 45 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.478123+08:00 NOTE: Corrupted block 46 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.495677+08:00 NOTE: Corrupted block 47 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.502943+08:00 NOTE: Corrupted block 48 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.512546+08:00 NOTE: Corrupted block 49 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.524469+08:00 NOTE: Corrupted block 50 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.535854+08:00 NOTE: Corrupted block 51 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.547352+08:00 NOTE: Corrupted block 52 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.558426+08:00 NOTE: Corrupted block 53 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.570391+08:00 NOTE: Corrupted block 54 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.582894+08:00 NOTE: Corrupted block 55 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.595051+08:00 NOTE: Corrupted block 56 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.606211+08:00 NOTE: Corrupted block 57 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.616777+08:00 NOTE: Corrupted block 58 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.629367+08:00 NOTE: Corrupted block 59 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.640406+08:00 NOTE: Corrupted block 60 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.651861+08:00 NOTE: Corrupted block 61 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.662557+08:00 NOTE: Corrupted block 62 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.674682+08:00 NOTE: Corrupted block 63 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.688506+08:00 NOTE: Corrupted block 64 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.699040+08:00 NOTE: Corrupted block 65 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.710109+08:00 NOTE: Corrupted block 66 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.723042+08:00 NOTE: Corrupted block 67 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.733776+08:00 NOTE: Corrupted block 68 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.743477+08:00 NOTE: Corrupted block 69 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.758913+08:00 NOTE: Corrupted block 70 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.768122+08:00 NOTE: Corrupted block 71 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.779376+08:00 NOTE: Corrupted block 72 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.790234+08:00 NOTE: Corrupted block 73 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.801020+08:00 NOTE: Corrupted block 74 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.813335+08:00 NOTE: Corrupted block 75 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.824036+08:00 NOTE: Corrupted block 76 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.835880+08:00 NOTE: Corrupted block 77 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.846727+08:00 NOTE: Corrupted block 78 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.866480+08:00 NOTE: Corrupted block 79 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.876746+08:00 NOTE: Corrupted block 80 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.886554+08:00 NOTE: Corrupted block 81 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.896282+08:00 NOTE: Corrupted block 82 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.908670+08:00 NOTE: Corrupted block 83 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.920153+08:00 NOTE: Corrupted block 84 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.931526+08:00 NOTE: Corrupted block 85 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.941594+08:00 NOTE: Corrupted block 86 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.953786+08:00 NOTE: Corrupted block 87 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.965813+08:00 NOTE: Corrupted block 88 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.979231+08:00 NOTE: Corrupted block 89 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:55.991114+08:00 NOTE: Corrupted block 90 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.002781+08:00 NOTE: Corrupted block 91 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.015067+08:00 NOTE: Corrupted block 92 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.026927+08:00 NOTE: Corrupted block 93 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.038930+08:00 NOTE: Corrupted block 94 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.058743+08:00 NOTE: Corrupted block 95 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.066922+08:00 NOTE: Corrupted block 96 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.075452+08:00 NOTE: Corrupted block 97 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.085976+08:00 NOTE: Corrupted block 98 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.097252+08:00 NOTE: Corrupted block 99 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.109468+08:00 NOTE: Corrupted block 100 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.121672+08:00 NOTE: Corrupted block 101 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.133366+08:00 NOTE: Corrupted block 102 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.143867+08:00 NOTE: Corrupted block 103 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.157345+08:00 NOTE: Corrupted block 104 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.167265+08:00 NOTE: Corrupted block 105 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.179456+08:00 NOTE: Corrupted block 106 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.189954+08:00 NOTE: Corrupted block 107 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.201496+08:00 NOTE: Corrupted block 108 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.214012+08:00 NOTE: Corrupted block 109 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.225167+08:00 NOTE: Corrupted block 110 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.238002+08:00 NOTE: Corrupted block 111 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.250434+08:00 NOTE: Corrupted block 112 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.261909+08:00 NOTE: Corrupted block 113 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.271939+08:00 NOTE: Corrupted block 114 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.294698+08:00 NOTE: Corrupted block 115 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.310502+08:00 NOTE: Corrupted block 116 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.317751+08:00 NOTE: Corrupted block 117 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.327346+08:00 NOTE: Corrupted block 118 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.336224+08:00 NOTE: Corrupted block 119 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.347744+08:00 NOTE: Corrupted block 120 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.358391+08:00 NOTE: Corrupted block 121 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.375956+08:00 NOTE: Corrupted block 122 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.410751+08:00 NOTE: Corrupted block 123 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.412970+08:00 NOTE: Corrupted block 124 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.415159+08:00 NOTE: Corrupted block 125 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.424999+08:00 NOTE: Corrupted block 126 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.440154+08:00 NOTE: Corrupted block 127 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.486039+08:00 NOTE: Corrupted block 128 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T11:42:56.530363+08:00 NOTE: Scrubbing finished 2019-06-26T11:42:56.530855+08:00 SUCCESS: alter diskgroup SCRUB scrub file '+SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf' wait |
下面开始实际修复这个文件,可以很清晰的看到Scrubbing用slave方式修复了刚刚报告的那些块。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 |
SQL> alter diskgroup SCRUB scrub file '+SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf' repair wait 2019-06-26T12:10:31.989817+08:00 NOTE: Waiting for scrubbing to finish 2019-06-26T12:10:32.084333+08:00 NOTE: Corrupted block 1 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:32.159155+08:00 NOTE: Corrupted block 2 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:32.177104+08:00 NOTE: Corrupted block 3 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:32.196471+08:00 NOTE: Scrubbing repair block 1 in file 256.1011956859 in slave 2019-06-26T12:10:32.214356+08:00 NOTE: Corrupted block 4 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:32.327877+08:00 NOTE: Scrubbing repair block 2 in file 256.1011956859 in slave 2019-06-26T12:10:32.329245+08:00 NOTE: Corrupted block 5 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:32.400884+08:00 NOTE: Scrubbing repair block 4 in file 256.1011956859 in slave 2019-06-26T12:10:32.410708+08:00 NOTE: Scrubbing repair block 3 in file 256.1011956859 in slave 2019-06-26T12:10:32.418532+08:00 NOTE: Corrupted block 7 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:32.476979+08:00 NOTE: Scrubbing repair block 5 in file 256.1011956859 in slave 2019-06-26T12:10:32.490964+08:00 NOTE: Scrubbing repair block 7 in file 256.1011956859 in slave 2019-06-26T12:10:32.539484+08:00 NOTE: Corrupted block 8 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:32.542973+08:00 NOTE: Corrupted block 9 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:32.543186+08:00 NOTE: Corrupted block 16 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:32.553290+08:00 NOTE: Scrubbing repair block 8 in file 256.1011956859 in slave 2019-06-26T12:10:32.558595+08:00 NOTE: Corrupted block 6 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:32.639307+08:00 NOTE: Scrubbing repair block 9 in file 256.1011956859 in slave 2019-06-26T12:10:32.667336+08:00 NOTE: Corrupted block 17 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:32.669144+08:00 NOTE: Corrupted block 18 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:32.681046+08:00 NOTE: Corrupted block 19 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:32.747537+08:00 NOTE: Scrubbing repair block 16 in file 256.1011956859 in slave 2019-06-26T12:10:32.841225+08:00 NOTE: Scrubbing repair block 17 in file 256.1011956859 in slave 2019-06-26T12:10:32.854281+08:00 NOTE: Corrupted block 21 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:32.854465+08:00 NOTE: Scrubbing repair block 6 in file 256.1011956859 in slave 2019-06-26T12:10:32.940134+08:00 NOTE: Scrubbing repair block 18 in file 256.1011956859 in slave 2019-06-26T12:10:32.952729+08:00 NOTE: Scrubbing repair block 19 in file 256.1011956859 in slave 2019-06-26T12:10:32.964025+08:00 NOTE: Scrubbing repair block 21 in file 256.1011956859 in slave 2019-06-26T12:10:33.027436+08:00 NOTE: Corrupted block 25 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:33.029294+08:00 NOTE: Corrupted block 23 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:33.031238+08:00 NOTE: Corrupted block 24 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:33.034072+08:00 NOTE: Corrupted block 22 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:33.035565+08:00 NOTE: Corrupted block 26 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:33.042291+08:00 NOTE: Corrupted block 20 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:33.104314+08:00 NOTE: Scrubbing repair block 25 in file 256.1011956859 in slave 2019-06-26T12:10:33.115875+08:00 NOTE: Scrubbing repair block 23 in file 256.1011956859 in slave 2019-06-26T12:10:33.129269+08:00 NOTE: Scrubbing repair block 24 in file 256.1011956859 in slave 2019-06-26T12:10:33.214821+08:00 NOTE: Corrupted block 30 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:33.215359+08:00 NOTE: Corrupted block 28 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:33.217090+08:00 NOTE: Corrupted block 29 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:33.238562+08:00 NOTE: Scrubbing repair block 22 in file 256.1011956859 in slave 2019-06-26T12:10:33.245916+08:00 NOTE: Corrupted block 27 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:33.252306+08:00 NOTE: Scrubbing repair block 20 in file 256.1011956859 in slave 2019-06-26T12:10:33.254739+08:00 NOTE: Scrubbing repair block 26 in file 256.1011956859 in slave 2019-06-26T12:10:33.330720+08:00 NOTE: Scrubbing repair block 30 in file 256.1011956859 in slave 2019-06-26T12:10:33.342046+08:00 NOTE: Scrubbing repair block 28 in file 256.1011956859 in slave 2019-06-26T12:10:33.354035+08:00 NOTE: Scrubbing repair block 29 in file 256.1011956859 in slave 2019-06-26T12:10:33.366707+08:00 NOTE: Scrubbing repair block 27 in file 256.1011956859 in slave 2019-06-26T12:10:34.310890+08:00 NOTE: Corrupted block 38 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:34.312275+08:00 NOTE: Corrupted block 34 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:34.312647+08:00 NOTE: Corrupted block 33 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:34.312952+08:00 NOTE: Corrupted block 37 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:34.313466+08:00 NOTE: Corrupted block 35 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:34.313210+08:00 NOTE: Corrupted block 36 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:34.314017+08:00 NOTE: Scrubbing repair block 38 in file 256.1011956859 in slave 2019-06-26T12:10:34.315861+08:00 NOTE: Corrupted block 32 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:34.329393+08:00 NOTE: Corrupted block 39 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:34.332587+08:00 NOTE: Scrubbing repair block 34 in file 256.1011956859 in slave 2019-06-26T12:10:34.334908+08:00 NOTE: Corrupted block 40 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:34.354628+08:00 NOTE: Scrubbing repair block 33 in file 256.1011956859 in slave 2019-06-26T12:10:34.476919+08:00 NOTE: Corrupted block 41 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:34.484534+08:00 NOTE: Corrupted block 43 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:34.484921+08:00 NOTE: Corrupted block 42 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:34.502293+08:00 NOTE: Scrubbing repair block 37 in file 256.1011956859 in slave 2019-06-26T12:10:34.507586+08:00 NOTE: Corrupted block 31 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:34.560348+08:00 NOTE: Scrubbing repair block 40 in file 256.1011956859 in slave 2019-06-26T12:10:34.566252+08:00 NOTE: Scrubbing repair block 35 in file 256.1011956859 in slave 2019-06-26T12:10:34.646183+08:00 NOTE: Scrubbing repair block 41 in file 256.1011956859 in slave 2019-06-26T12:10:34.651316+08:00 NOTE: Scrubbing repair block 36 in file 256.1011956859 in slave 2019-06-26T12:10:34.720312+08:00 NOTE: Scrubbing repair block 32 in file 256.1011956859 in slave 2019-06-26T12:10:34.734270+08:00 NOTE: Scrubbing repair block 42 in file 256.1011956859 in slave 2019-06-26T12:10:34.746236+08:00 NOTE: Scrubbing repair block 39 in file 256.1011956859 in slave 2019-06-26T12:10:34.748061+08:00 NOTE: Scrubbing repair block 43 in file 256.1011956859 in slave 2019-06-26T12:10:34.768308+08:00 NOTE: Scrubbing repair block 31 in file 256.1011956859 in slave 2019-06-26T12:10:35.587676+08:00 NOTE: Corrupted block 53 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:35.587954+08:00 NOTE: Corrupted block 49 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:35.588032+08:00 NOTE: Corrupted block 52 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:35.589980+08:00 NOTE: Corrupted block 51 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:35.590797+08:00 NOTE: Scrubbing repair block 53 in file 256.1011956859 in slave 2019-06-26T12:10:35.591771+08:00 NOTE: Corrupted block 50 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:35.594156+08:00 NOTE: Corrupted block 48 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:35.596365+08:00 NOTE: Corrupted block 47 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:35.598362+08:00 NOTE: Corrupted block 46 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:35.602076+08:00 NOTE: Corrupted block 45 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:35.604271+08:00 NOTE: Corrupted block 44 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:35.606857+08:00 NOTE: Scrubbing repair block 49 in file 256.1011956859 in slave 2019-06-26T12:10:35.738535+08:00 NOTE: Scrubbing repair block 52 in file 256.1011956859 in slave 2019-06-26T12:10:35.743017+08:00 NOTE: Corrupted block 55 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:35.745714+08:00 NOTE: Corrupted block 54 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:35.748166+08:00 NOTE: Corrupted block 56 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:35.751244+08:00 NOTE: Scrubbing repair block 51 in file 256.1011956859 in slave 2019-06-26T12:10:35.878787+08:00 NOTE: Corrupted block 60 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:35.878987+08:00 NOTE: Corrupted block 58 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:35.880115+08:00 NOTE: Corrupted block 57 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:35.883858+08:00 NOTE: Corrupted block 59 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:35.886227+08:00 NOTE: Scrubbing repair block 45 in file 256.1011956859 in slave 2019-06-26T12:10:35.898224+08:00 NOTE: Scrubbing repair block 50 in file 256.1011956859 in slave 2019-06-26T12:10:35.910471+08:00 NOTE: Scrubbing repair block 44 in file 256.1011956859 in slave 2019-06-26T12:10:35.922885+08:00 NOTE: Scrubbing repair block 48 in file 256.1011956859 in slave 2019-06-26T12:10:36.037346+08:00 NOTE: Corrupted block 65 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.039816+08:00 NOTE: Corrupted block 68 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.041538+08:00 NOTE: Scrubbing repair block 58 in file 256.1011956859 in slave 2019-06-26T12:10:36.042782+08:00 NOTE: Corrupted block 66 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.044554+08:00 NOTE: Corrupted block 64 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.046365+08:00 NOTE: Corrupted block 67 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.047939+08:00 NOTE: Corrupted block 63 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.049531+08:00 NOTE: Corrupted block 62 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.051281+08:00 NOTE: Corrupted block 61 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.059007+08:00 NOTE: Scrubbing repair block 47 in file 256.1011956859 in slave 2019-06-26T12:10:36.071099+08:00 NOTE: Scrubbing repair block 57 in file 256.1011956859 in slave 2019-06-26T12:10:36.090157+08:00 NOTE: Scrubbing repair block 46 in file 256.1011956859 in slave 2019-06-26T12:10:36.094352+08:00 NOTE: Corrupted block 70 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.097997+08:00 NOTE: Corrupted block 69 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.104341+08:00 NOTE: Corrupted block 71 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.106891+08:00 NOTE: Scrubbing repair block 59 in file 256.1011956859 in slave 2019-06-26T12:10:36.120432+08:00 NOTE: Scrubbing repair block 55 in file 256.1011956859 in slave 2019-06-26T12:10:36.122946+08:00 NOTE: Corrupted block 74 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.123958+08:00 NOTE: Corrupted block 72 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.126522+08:00 NOTE: Corrupted block 73 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.134085+08:00 NOTE: Scrubbing repair block 54 in file 256.1011956859 in slave 2019-06-26T12:10:36.138949+08:00 NOTE: Corrupted block 75 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.147412+08:00 NOTE: Scrubbing repair block 56 in file 256.1011956859 in slave 2019-06-26T12:10:36.148223+08:00 NOTE: Corrupted block 76 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.158682+08:00 NOTE: Corrupted block 77 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.163501+08:00 NOTE: Scrubbing repair block 68 in file 256.1011956859 in slave 2019-06-26T12:10:36.173124+08:00 NOTE: Corrupted block 78 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.175013+08:00 NOTE: Scrubbing repair block 60 in file 256.1011956859 in slave 2019-06-26T12:10:36.195354+08:00 NOTE: Scrubbing repair block 66 in file 256.1011956859 in slave 2019-06-26T12:10:36.196567+08:00 NOTE: Corrupted block 79 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.206864+08:00 NOTE: Scrubbing repair block 65 in file 256.1011956859 in slave 2019-06-26T12:10:36.209751+08:00 NOTE: Corrupted block 80 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.218787+08:00 NOTE: Scrubbing repair block 64 in file 256.1011956859 in slave 2019-06-26T12:10:36.221233+08:00 NOTE: Corrupted block 81 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.230710+08:00 NOTE: Scrubbing repair block 67 in file 256.1011956859 in slave 2019-06-26T12:10:36.241188+08:00 NOTE: Corrupted block 82 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.243091+08:00 NOTE: Scrubbing repair block 63 in file 256.1011956859 in slave 2019-06-26T12:10:36.245431+08:00 NOTE: Corrupted block 83 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.253570+08:00 NOTE: Scrubbing repair block 62 in file 256.1011956859 in slave 2019-06-26T12:10:36.262262+08:00 NOTE: Corrupted block 84 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.265499+08:00 NOTE: Scrubbing repair block 61 in file 256.1011956859 in slave 2019-06-26T12:10:36.276813+08:00 NOTE: Scrubbing repair block 70 in file 256.1011956859 in slave 2019-06-26T12:10:36.277913+08:00 NOTE: Corrupted block 85 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.287478+08:00 NOTE: Corrupted block 86 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.288851+08:00 NOTE: Scrubbing repair block 69 in file 256.1011956859 in slave 2019-06-26T12:10:36.297127+08:00 NOTE: Corrupted block 87 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.300284+08:00 NOTE: Scrubbing repair block 71 in file 256.1011956859 in slave 2019-06-26T12:10:36.308145+08:00 NOTE: Corrupted block 88 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.312572+08:00 NOTE: Scrubbing repair block 74 in file 256.1011956859 in slave 2019-06-26T12:10:36.322344+08:00 NOTE: Corrupted block 89 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.323668+08:00 NOTE: Scrubbing repair block 72 in file 256.1011956859 in slave 2019-06-26T12:10:36.334546+08:00 NOTE: Corrupted block 90 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.335800+08:00 NOTE: Scrubbing repair block 73 in file 256.1011956859 in slave 2019-06-26T12:10:36.348095+08:00 NOTE: Scrubbing repair block 75 in file 256.1011956859 in slave 2019-06-26T12:10:36.360738+08:00 NOTE: Scrubbing repair block 76 in file 256.1011956859 in slave 2019-06-26T12:10:36.363519+08:00 NOTE: Corrupted block 91 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.376459+08:00 NOTE: Scrubbing repair block 77 in file 256.1011956859 in slave 2019-06-26T12:10:36.387100+08:00 NOTE: Corrupted block 92 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.394800+08:00 NOTE: Scrubbing repair block 78 in file 256.1011956859 in slave 2019-06-26T12:10:36.403125+08:00 NOTE: Corrupted block 93 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.408485+08:00 NOTE: Corrupted block 94 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.415315+08:00 NOTE: Scrubbing repair block 79 in file 256.1011956859 in slave 2019-06-26T12:10:36.421784+08:00 NOTE: Corrupted block 95 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.421776+08:00 NOTE: Corrupted block 96 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.428690+08:00 NOTE: Scrubbing repair block 80 in file 256.1011956859 in slave 2019-06-26T12:10:36.435058+08:00 NOTE: Corrupted block 97 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.437001+08:00 NOTE: Corrupted block 98 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.440111+08:00 NOTE: Scrubbing repair block 81 in file 256.1011956859 in slave 2019-06-26T12:10:36.451997+08:00 NOTE: Scrubbing repair block 82 in file 256.1011956859 in slave 2019-06-26T12:10:36.457870+08:00 NOTE: Corrupted block 99 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.463574+08:00 NOTE: Scrubbing repair block 83 in file 256.1011956859 in slave 2019-06-26T12:10:36.466654+08:00 NOTE: Corrupted block 100 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.475400+08:00 NOTE: Scrubbing repair block 84 in file 256.1011956859 in slave 2019-06-26T12:10:36.478663+08:00 NOTE: Corrupted block 101 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.489790+08:00 NOTE: Scrubbing repair block 85 in file 256.1011956859 in slave 2019-06-26T12:10:36.493224+08:00 NOTE: Corrupted block 102 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.500731+08:00 NOTE: Scrubbing repair block 86 in file 256.1011956859 in slave 2019-06-26T12:10:36.501906+08:00 NOTE: Corrupted block 103 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.508870+08:00 NOTE: Corrupted block 104 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.513921+08:00 NOTE: Scrubbing repair block 87 in file 256.1011956859 in slave 2019-06-26T12:10:36.525790+08:00 NOTE: Scrubbing repair block 88 in file 256.1011956859 in slave 2019-06-26T12:10:36.529155+08:00 NOTE: Corrupted block 105 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.536896+08:00 NOTE: Scrubbing repair block 89 in file 256.1011956859 in slave 2019-06-26T12:10:36.545533+08:00 NOTE: Corrupted block 106 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.550426+08:00 NOTE: Scrubbing repair block 90 in file 256.1011956859 in slave 2019-06-26T12:10:36.552522+08:00 NOTE: Corrupted block 107 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.562056+08:00 NOTE: Scrubbing repair block 91 in file 256.1011956859 in slave 2019-06-26T12:10:36.569162+08:00 NOTE: Corrupted block 108 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.573778+08:00 NOTE: Scrubbing repair block 92 in file 256.1011956859 in slave 2019-06-26T12:10:36.583868+08:00 NOTE: Corrupted block 109 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.585840+08:00 NOTE: Scrubbing repair block 93 in file 256.1011956859 in slave 2019-06-26T12:10:36.597118+08:00 NOTE: Corrupted block 110 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.598678+08:00 NOTE: Scrubbing repair block 94 in file 256.1011956859 in slave 2019-06-26T12:10:36.609746+08:00 NOTE: Scrubbing repair block 95 in file 256.1011956859 in slave 2019-06-26T12:10:36.612062+08:00 NOTE: Corrupted block 111 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.621672+08:00 NOTE: Corrupted block 112 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.622139+08:00 NOTE: Scrubbing repair block 96 in file 256.1011956859 in slave 2019-06-26T12:10:36.627646+08:00 NOTE: Corrupted block 113 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.634337+08:00 NOTE: Scrubbing repair block 97 in file 256.1011956859 in slave 2019-06-26T12:10:36.641882+08:00 NOTE: Corrupted block 114 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.645118+08:00 NOTE: Scrubbing repair block 98 in file 256.1011956859 in slave 2019-06-26T12:10:36.650516+08:00 NOTE: Corrupted block 115 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.658322+08:00 NOTE: Scrubbing repair block 99 in file 256.1011956859 in slave 2019-06-26T12:10:36.670048+08:00 NOTE: Scrubbing repair block 100 in file 256.1011956859 in slave 2019-06-26T12:10:36.673525+08:00 NOTE: Corrupted block 117 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.675119+08:00 NOTE: Corrupted block 116 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.682070+08:00 NOTE: Scrubbing repair block 101 in file 256.1011956859 in slave 2019-06-26T12:10:36.689478+08:00 NOTE: Corrupted block 118 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.693754+08:00 NOTE: Scrubbing repair block 102 in file 256.1011956859 in slave 2019-06-26T12:10:36.706352+08:00 NOTE: Scrubbing repair block 103 in file 256.1011956859 in slave 2019-06-26T12:10:36.704577+08:00 NOTE: Corrupted block 119 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.718454+08:00 NOTE: Scrubbing repair block 104 in file 256.1011956859 in slave 2019-06-26T12:10:36.722048+08:00 NOTE: Corrupted block 120 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.730684+08:00 NOTE: Scrubbing repair block 105 in file 256.1011956859 in slave 2019-06-26T12:10:36.736762+08:00 NOTE: Corrupted block 121 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.743205+08:00 NOTE: Scrubbing repair block 106 in file 256.1011956859 in slave 2019-06-26T12:10:36.747836+08:00 NOTE: Corrupted block 122 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.751198+08:00 NOTE: Corrupted block 123 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.754756+08:00 NOTE: Scrubbing repair block 107 in file 256.1011956859 in slave 2019-06-26T12:10:36.761567+08:00 NOTE: Corrupted block 124 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.766799+08:00 NOTE: Scrubbing repair block 108 in file 256.1011956859 in slave 2019-06-26T12:10:36.772027+08:00 NOTE: Corrupted block 125 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.779018+08:00 NOTE: Scrubbing repair block 109 in file 256.1011956859 in slave 2019-06-26T12:10:36.785160+08:00 NOTE: Corrupted block 126 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.790884+08:00 NOTE: Scrubbing repair block 110 in file 256.1011956859 in slave 2019-06-26T12:10:36.796411+08:00 NOTE: Corrupted block 127 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.801715+08:00 NOTE: Scrubbing repair block 111 in file 256.1011956859 in slave 2019-06-26T12:10:36.807755+08:00 NOTE: Corrupted block 128 found in group 3 file +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf 2019-06-26T12:10:36.813814+08:00 NOTE: Scrubbing repair block 112 in file 256.1011956859 in slave 2019-06-26T12:10:36.825948+08:00 NOTE: Scrubbing repair block 113 in file 256.1011956859 in slave 2019-06-26T12:10:36.836439+08:00 NOTE: Scrubbing repair block 114 in file 256.1011956859 in slave 2019-06-26T12:10:36.848007+08:00 NOTE: Scrubbing repair block 115 in file 256.1011956859 in slave 2019-06-26T12:10:36.859723+08:00 NOTE: Scrubbing repair block 117 in file 256.1011956859 in slave 2019-06-26T12:10:36.871642+08:00 NOTE: Scrubbing repair block 116 in file 256.1011956859 in slave 2019-06-26T12:10:36.883666+08:00 NOTE: Scrubbing repair block 118 in file 256.1011956859 in slave 2019-06-26T12:10:36.894869+08:00 NOTE: Scrubbing repair block 119 in file 256.1011956859 in slave 2019-06-26T12:10:36.906784+08:00 NOTE: Scrubbing repair block 120 in file 256.1011956859 in slave 2019-06-26T12:10:36.918508+08:00 NOTE: Scrubbing repair block 121 in file 256.1011956859 in slave 2019-06-26T12:10:36.930415+08:00 NOTE: Scrubbing repair block 122 in file 256.1011956859 in slave 2019-06-26T12:10:36.941412+08:00 NOTE: Scrubbing repair block 123 in file 256.1011956859 in slave 2019-06-26T12:10:36.952655+08:00 NOTE: Scrubbing repair block 124 in file 256.1011956859 in slave 2019-06-26T12:10:36.964396+08:00 NOTE: Scrubbing repair block 125 in file 256.1011956859 in slave 2019-06-26T12:10:36.976429+08:00 NOTE: Scrubbing repair block 126 in file 256.1011956859 in slave 2019-06-26T12:10:36.987416+08:00 NOTE: Scrubbing repair block 127 in file 256.1011956859 in slave 2019-06-26T12:10:36.999391+08:00 NOTE: Scrubbing repair block 128 in file 256.1011956859 in slave 2019-06-26T12:10:37.120417+08:00 NOTE: Scrubbing finished 2019-06-26T12:10:37.120823+08:00 SUCCESS: alter diskgroup SCRUB scrub file '+SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf' repair wait |
继续把这个文件拿出来,做dbv校验:
1 2 3 4 5 6 7 8 |
[grid@dm01db08 scrub]$ dbv file=1D26AU.dbf DBVERIFY: Release 19.0.0.0.0 - Production on Wed Jun 26 12:34:40 2019 Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved. DBV-00107: Unknown header format (0) (0) |
bbed打开这个文件看:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
[ora19c@dm01db08 scrub]$ bbed password=blockedit filename=/tmp/1D26AU.dbf blocksize=8192 mode=edit BBED: Release 2.0.0.0.0 - Limited Production on Wed Jun 26 12:35:42 2019 Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved. ************* !!! For Oracle Internal Use only !!! *************** BBED> show FILE# 0 BLOCK# 1 OFFSET 0 DBA 0x00000000 (0 0,1) FILENAME /tmp/1D26AU.dbf BIFILE bifile.bbd LISTFILE BLOCKSIZE 512 MODE Edit EDIT Unrecoverable IBASE Dec OBASE Dec WIDTH 80 COUNT 512 LOGFILE log.bbd SPOOL No BBED> map File: /tmp/1D26AU.dbf (0) Block: 1 Dba:0x00000000 ------------------------------------------------------------ BBED-00400: invalid blocktype (00) |
而如果文件正常的话,打开应该是下面这个样子的:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
BBED> set block 1 BLOCK# 1 BBED> map File: /tmp/0D26AU.dbf (0) Block: 1 Dba:0x00000000 ------------------------------------------------------------ Data File Header struct kcvfh, 1272 bytes @0 ub4 tailchk @8188 BBED> p kcvfh struct kcvfh, 1272 bytes @0 struct kcvfhbfh, 20 bytes @0 ub1 type_kcbh @0 0x0b ub1 frmt_kcbh @1 0xa2 ub2 wrp2_kcbh @2 0x0000 ub4 rdba_kcbh @4 0x0c800001 ub4 bas_kcbh @8 0x00000000 ub2 wrp_kcbh @12 0x0000 ub1 seq_kcbh @14 0x01 ub1 flg_kcbh @15 0x04 (KCBHFCKV) ub2 chkval_kcbh @16 0x5430 ub2 spare3_kcbh @18 0x0000 struct kcvfhhdr, 76 bytes @20 ub4 kccfhswv @20 0x00000000 ub4 kccfhcvn @24 0x13000000 ub4 kccfhdbi @28 0x3f58538b text kccfhdbn[0] @32 O text kccfhdbn[1] @33 R text kccfhdbn[2] @34 A text kccfhdbn[3] @35 1 text kccfhdbn[4] @36 9 text kccfhdbn[5] @37 C text kccfhdbn[6] @38 text kccfhdbn[7] @39 ub4 kccfhcsq @40 0x0001545f ub4 kccfhfsz @44 0x00000080 s_blkz kccfhbsz @48 0x00 ub2 kccfhfno @52 0x0032 ub2 kccfhtyp @54 0x0003 ub4 kccfhacid @56 0x00000000 ub4 kccfhcks @60 0x00000000 |
但是Scrubbing有三种方式,基于文件,基于磁盘,基于磁盘组,据测试,三种方式都不可以完成头部块的修复。
到此,前面两个场景的测试就完成了。效果是有,但是并不是太好。
最后一个场景,用bbed手工修改其中一个AU的某行数据。让两个AU里某行数据不一样,但是确保dbv测试通过的情况下,ASM Disk Scrubbing会工作吗?
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 |
SQL> select * from tb_scrub; ID NAME ---------- ------------------------------------------------------------ 1 MINOR 2 MINOR select EXTENT_ID, BLOCK_ID, BLOCKS, FILE_ID from dba_extents where SEGMENT_NAME='TB_SCRUB' and OWNER='SYS'; EXTENT_ID BLOCK_ID BLOCKS FILE_ID ---------- ---------- ---------- ---------- 0 8 8 51 select name from v$datafile where file#=51; NAME ---------------------------------------------------- +SCRUB/ORA19c/ORA19PDB1/scrub_001.dbf select GROUP_NUMBER from V$ASM_DISKGROUP where NAME like '%SCRUB%'; GROUP_NUMBER ------------ 3 在ASM用户中查询相关数据 --ASM中执行 SQL> select PXN_KFFXP, -- physical extent number \ 2 XNUM_KFFXP, -- virtual extent number 3 DISK_KFFXP, -- disk number 4 AU_KFFXP, -- allocation unit number 5 decode(LXN_KFFXP,0,'Primary',1,'Secondary','header metadata') "AU type" 6 from X$KFFXP 7 where NUMBER_KFFXP=256 -- ASM file 272 8 AND GROUP_KFFXP=3 -- group number 1 9 order by 1; PXN_KFFXP XNUM_KFFXP DISK_KFFXP AU_KFFXP AU type ---------- ---------- ---------- ---------- --------------------------------------------- 0 0 1 26 Primary 1 0 0 26 Secondary SQL> SQL> SQL> SELECT disk_kffxp, au_kffxp, xnum_kffxp 2 FROM x$kffxp 3 WHERE GROUP_KFFXP=3 4 AND NUMBER_KFFXP=256; DISK_KFFXP AU_KFFXP XNUM_KFFXP ---------- ---------- ---------- 1 26 0 0 26 0 |
上面这段,确认了primary copy的位置。我们一会儿就要改这个copy。
可以得出该表TB_SCRUB的数据分布在第26个AU中(DISK_KFFXP=1/AU_KFFXP=26/XNUM_KFFXP=0)
是块8,9,10,11,12,13,14,15这些块。一共64kb。
验证如下:
dd if=/dev/mapper/data05 bs=8k count=8 skip=13320|strings| more
能看到表中的数据MINOR和MINOR。
1 2 |
dd if=/dev/mapper/data05 bs=8192 count=8 skip=13320 of=/tmp/minor.tab - -26*4*1024/8+8就是要跳过的13320个块 |
然后我们用bbed修改这个minor.tab文件,这里要感谢翔宇解惑,他有很方便的方式来修改。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
bbed password=blockedit filename=/tmp/minor.tab blocksize=8192 mode=edit - - BBED> set block 6 BLOCK# 6 BBED> p *kdbr[0] rowdata[12] ----------- ub1 rowdata[12] @8176 0x2c BBED> x /rn rowdata[12] @8176 ----------- flag@8176: 0x2c (KDRHFL, KDRHFF, KDRHFH) lock@8177: 0x01 cols@8178: 2 col 0[2] @8179: 1 col 1[5] @8182: ######################################### BBED> x /rc rowdata[12] @8176 ----------- flag@8176: 0x2c (KDRHFL, KDRHFF, KDRHFH) lock@8177: 0x01 cols@8178: 2 col 0[2] @8179: .. col 1[5] @8182: MINOR - - 我修改其中一个大写的MINOR为小写 dump /v find /c MINOR m /c minor #modify /x 4D494E4F52 filename '/tmp/minor.tab' block 6. offset 8171. dump /v sum sum apply |
然后把这个minor.tab文件写回磁盘上。
1 2 3 4 5 |
[grid@dm01db08 scrub]$ dd of=/dev/mapper/data05 bs=8192 count=8 seek=13320 if=/tmp/minor.tab 8+0 records in 8+0 records out 65536 bytes (66 kB) copied, 0.000697375 s, 94.0 MB/s |
在数据库上flush buffer_cache之后,查询数据。
1 2 3 4 5 6 7 8 9 10 11 12 |
SQL> alter system flush buffer_cache; System altered. SQL> select * from tb_scrub; ID NAME ---------- ------------------------------------------------------------ 1 MINOR 2 minor 可以看到数据确实被修改掉了。 |
如果,把这个磁盘offline呢?因为它是primary copy,它offline了,ASM会去读secondary copy,会看到之前的数据吗?
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
SQL> alter diskgroup scrub offline disk scrub_0001; Diskgroup altered. --DB查询 SQL> alter system flush buffer_cache; System altered. SQL> select * from tb_scrub; ID NAME ---------- ------------------------------------------------------------ 1 MINOR 2 MINOR - - 确实读了secondary copy了。 |
下面是全文的重点,这两份mirror是不一致的,但是都是合法的,从dbv等等一切block check方式上都看不出来它们的逻辑错。这个时候用ASM Disk Scrubbing手工运行一下会发生什么?
1 2 3 4 5 6 7 8 |
2019-06-26T14:55:06.867354+08:00 SQL> ALTER DISKGROUP SCRUB SCRUB REPAIR POWER LOW 2019-06-26T14:55:06.869383+08:00 NOTE: Start scrubbing diskgroup SCRUB 2019-06-26T14:55:06.898174+08:00 SUCCESS: ALTER DISKGROUP SCRUB SCRUB REPAIR POWER LOW 命令发起了,但是scrub并没有去修复这个块。 |
这两个互相矛盾的镜像会共存。offline 掉primary,读的secondary copy跟primary还是依然不一样的。所以Scrubbing对这种场景下的讹误是不会操作的。
— END